tech stuff

Main - Archive - About - Feed

APT repo on github

Case: repack/merge several .deb packages into a new one, and distribute via APT repo, hosted on github pages.

Extract existing debian package:

$ wget -c http://repository.maemo.org/extras-devel/pool/fremantle/free/c/cryptsetup/cryptsetup_1.0.7-12maemo0_armel.deb
$ dpkg-deb -x cryptsetup_1.0.7-12maemo0_armel.deb cryptsetup
$ dpkg-deb -e cryptsetup_1.0.7-12maemo0_armel.deb cryptsetup/DEBIAN

After modifications (deoptifying in this case) and merging, editing control files in DEBIAN/ subdir (see Debian Policy Manual for details on Control files and their fields), build it to a new .deb:

$ dpkg-deb -b cryptsetup/ ./

In this case new package is created in current dir, named by fields in DEBIAN/control file:

dpkg-deb: building package 'cryptsetup-deopt' in './/cryptsetup-deopt_1.0.7-maemo0_armel.deb'.

You can define package filename instead of ./ or omit it (package is named by the dir then).

Create github repo for an APT repo (from the cmdline, using API, of course). Export github username and pass as variables. I do it by sourcing a file, containing

export GHU="<github_user>"
export GHP="<gitgub_password>"
$ source ./github_credentials.txt

Then we can use github API via curl to create github repo:

$ curl -u "${GHU}:${GHP}" https://api.github.com/user/repos -d '{"name": "apt-repo", "auto_init": true}'

where "apt-repo" stands for repository name. Make a fresh clone of newly created repo:

$ git clone git@github.com:${GHU}/apt-repo

Create a gh-pages branch, making it orphan, without any parents, and clean out its content:

$ cd apt-repo/
$ git checkout --orphan gh-pages
$ git rm -rf .

Add some dummy content and push:

$ echo "APT repo"  > index.html
$ git add index.html
$ git commit -a -m "Initial commit"
$ git push origin gh-pages

Install reprepro:

$ sudo apt-get install reprepro

Generate signing key for the repo:

$ gpg --gen-key

Configure reprepro as described here, and init APT repo, by adding previously repacked debfile to it:

$ mkdir conf
$ vi conf/distributions
$ cat <<EOF >> conf/options
verbose
basedir .
ask-passphrase
EOF
$ reprepro includedeb fremantle ../cryptsetup-nonopt_1.0.7maemo0_armel.deb

Export public key:

$ gpg --armor --output pubkey.gpg --export <key-id>

A final touch by adding some useful info about the repo to index.html (gitpages do not use directory listings and you must have index files, or you get 404 on directory).

$ vi index.html

Commit, push and enjoy:

$ git add --all
$ git commit -m "APT repo init"
$ git push origin gh-pages

Result is here: http://u0d7i.github.io/apt-repo/

DNS query with netcat

First, set up tcpdump for traffic capture to file and display to stdout at the same time:

$ tcpdump -n -U -w - -i gprs0 | tee /tmp/test.cap | tcpdump -n -r -

Execute 'host' query towards google public DNS:

$ host -t a google-public-dns-a.google.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

google-public-dns-a.google.com has address 8.8.8.8

tcpdump shows:

10:59:38.665865 IP 10.120.149.34.63989 > 8.8.8.8.53: 46669+ A? google-public-dns-a.google.com. (48)
10:59:40.886812 IP 8.8.8.8.53 > 10.120.149.34.63989: 46669 1/0/0 A 8.8.8.8 (64)

We can dig into the request packet with tshark (decode part skipped to DNS):

$ tshark -r /tmp/test.cap -V -x -R frame.number==1
...
Domain Name System (query)
    Transaction ID: 0xb64d
    Flags: 0x0100 Standard query
        0... .... .... .... = Response: Message is a query
        .000 0... .... .... = Opcode: Standard query (0)
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... .0.. .... = Z: reserved (0)
        .... .... ...0 .... = Non-authenticated data: Unacceptable
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 0
    Queries
        google-public-dns-a.google.com: type A, class IN
            Name: google-public-dns-a.google.com
            Type: A (Host address)
            Class: IN (0x0001)

0000  00 04 03 35 00 00 00 00 00 00 00 00 00 00 08 00   ...5............
0010  45 00 00 4c 0d 66 00 00 40 11 bd 91 0a 78 95 22   E..L.f..@....x."
0020  08 08 08 08 f9 f5 00 35 00 38 36 b4 b6 4d 01 00   .......5.86..M..
0030  00 01 00 00 00 00 00 00 13 67 6f 6f 67 6c 65 2d   .........google-
0040  70 75 62 6c 69 63 2d 64 6e 73 2d 61 06 67 6f 6f   public-dns-a.goo
0050  67 6c 65 03 63 6f 6d 00 00 01 00 01               gle.com.....

In hex dump we can see DNS payload starting with transaction ID "b6 4d" at 002c. With this sample we can easily encode our own DNS query using hex ASCII escape chars, and pass it to netcat (reply is piped to hexdump):

$ echo -n -e "\x13\x37\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x13google-public-dns-a\x06google\x03com\x00\x00\x01\x00\x01" | nc -u -w1 8.8.8.8 53 | hexdump -C
00000000  13 37 81 80 00 01 00 01  00 00 00 00 13 67 6f 6f  |.7...........goo|
00000010  67 6c 65 2d 70 75 62 6c  69 63 2d 64 6e 73 2d 61  |gle-public-dns-a|
00000020  06 67 6f 6f 67 6c 65 03  63 6f 6d 00 00 01 00 01  |.google.com.....|
00000030  c0 0c 00 01 00 01 00 00  53 b2 00 04 08 08 08 08  |........S.......|
00000040

tcpdump shows:

11:02:32.045931 IP 10.120.149.34.55204 > 8.8.8.8.53: 4919+ A? google-public-dns-a.google.com. (48)
11:02:33.750337 IP 8.8.8.8.53 > 10.120.149.34.55204: 4919 1/0/0 A 8.8.8.8 (64)

We see valid DNS session with ID 4919 (decimal for our 0x1337, crafted in first two octets of our string), and we can decode reply packet from server:

 tshark -r /tmp/test.cap -V -x -R frame.number==4
...
Domain Name System (response)
    [Request In: 3]
    [Time: 1.704406000 seconds]
    Transaction ID: 0x1337
    Flags: 0x8180 Standard query response, No error
        1... .... .... .... = Response: Message is a response
        .000 0... .... .... = Opcode: Standard query (0)
        .... .0.. .... .... = Authoritative: Server is not an authority for domain
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... 1... .... = Recursion available: Server can do recursive queries
        .... .... .0.. .... = Z: reserved (0)
        .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server
        .... .... ...0 .... = Non-authenticated data: Unacceptable
        .... .... .... 0000 = Reply code: No error (0)
    Questions: 1
    Answer RRs: 1
    Authority RRs: 0
    Additional RRs: 0
    Queries
        google-public-dns-a.google.com: type A, class IN
            Name: google-public-dns-a.google.com
            Type: A (Host address)
            Class: IN (0x0001)
    Answers
        google-public-dns-a.google.com: type A, class IN, addr 8.8.8.8
            Name: google-public-dns-a.google.com
            Type: A (Host address)
            Class: IN (0x0001)
            Time to live: 5 hours, 57 minutes, 6 seconds
            Data length: 4
            Addr: 8.8.8.8 (8.8.8.8)

0000  00 00 03 35 00 00 00 00 00 00 00 00 00 00 08 00   ...5............
0010  45 40 00 5c 7d c2 00 00 35 11 57 e5 08 08 08 08   E@.\}...5.W.....
0020  0a 78 95 22 00 35 d7 a4 00 48 57 a5 13 37 81 80   .x.".5...HW..7..
0030  00 01 00 01 00 00 00 00 13 67 6f 6f 67 6c 65 2d   .........google-
0040  70 75 62 6c 69 63 2d 64 6e 73 2d 61 06 67 6f 6f   public-dns-a.goo
0050  67 6c 65 03 63 6f 6d 00 00 01 00 01 c0 0c 00 01   gle.com.........
0060  00 01 00 00 53 b2 00 04 08 08 08 08               ....S.......

String contents in detail:

\x13\x37 Transaction ID
\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00 Flags for standard, recursive, 1-question query
\x13google-public-dns-a\x06google\x03com \x00 Query string, null-terminated, each section starts with length number
\x00\x01\x00\x01 Query type (A) and class (IN)

See RFC 1035.

RPi benchmarking

There is a bunch of different Raspberry Pi boards on my desk (B, B+, 2B):

B and B+ have same CPU, B+ and 2B looks the same. I wanted to compare performance, so I did some benchmarking - the old classic way, by compiling Linux kernel onboard. First, I needed reliable SD card (I/O thingie impacts results a lot). There were several available right away:

Did some tests (h2testw, F3, etc.):

Kingston 4G C4
Writing speed: 4.07 MByte/s
Reading speed: 11.6 MByte/s
Kingston 4G C10
Writing speed: 8.73 MByte/s
Reading speed: 11.7 MByte/s
SanDisk Ultra 16G U1
Writing speed: 4.95 MByte/s
Reading speed: 11.7 MByte/s
Sandisk Ultra 8G C10
Writing speed: 11.0 MByte/s
Reading speed: 11.7 MByte/s
Samsung EVO 8G U1
Writing speed: 8.39 MByte/s
Reading speed: 11.7 MByte/s

Sandisk Ultra 8G C10 was the best for a job, imaged latest Raspbian on it. Test prep:

$ sudo apt-get install bc
$ mkdir perftest
$ cd perftest
$ wget -c https://www.kernel.org/pub/linux/kernel/v3.x/linux-$(uname -r | cut -d- -f1).tar.xz
$ tar -xvf linux*.xz

Performed the same test with the same SD card on 3 RPi Boards:

$ cd linux*/
$ rm -f ../*.deb
$ make mrproper
$ zcat /proc/config.gz > .config
$ yes "" | make oldconfig

Used gpio utility from Wiring Pi to id the board, and timed 'debian way' of compiling kernel to benchmark:

$ gpio -v | tail -2
$ time make -j$(nproc) deb-pkg

Results:

$ gpio -v | tail -2
Raspberry Pi Details:
  Type: Model B, Revision: 2, Memory: 512MB, Maker: Sony

$ time make -j$(nproc) deb-pkg
...
real    741m37.328s
user    693m9.250s
sys     31m57.460s
$ gpio -v | tail -2
Raspberry Pi Details:
  Type: Model B+, Revision: 1.2, Memory: 512MB, Maker: Sony

$ time make -j$(nproc) deb-pkg
...
real    741m18.997s
user    692m32.580s
sys     32m7.080s
$ gpio -v | tail -2
Raspberry Pi Details:
  Type: Model 2, Revision: 1.1, Memory: 1024MB, Maker: Sony

$ time make -j$(nproc) deb-pkg
...
real    111m51.491s
user    377m23.940s
sys     23m14.370s

Summary:

B/B+: 12 hours 21 minutes
2B: 1 hour 52 minutes

HTC One (M8) dual sim

I own Nokia E71 as my primary phone since 2009. In 6 years I've replaced LCD twice (once in a field pub), screen glass twice, and full cover housing once.

Now it's time for a change.

The new phone happened to be HTC One (M8) dual sim variant (for practical reasons I prefer last but one versions of everything).

Links:

Phone uses nano-SIM cards - I had to cut my regular SIMs.

I expected to run CyanogenMod on it, but found out dual-sim variant is not supported.

Before starting doing anithing with the device, you should read at least two XDA forum threads:

HTC One (M8) comes with locked bootloader, but HTC is kind enough to provide an official way to unlock it (you need to register on htcdev.com, but John Doe with a disposable e-mail goes fine). The process itself and related topics, like ROM flashing and building custom kernels are documented there as well, so it's worth read.

Some info about the phone in stock condition:
Model number on the case:

Model: 0P6B640 M8e

Software info from OS:

Amdroid version: 4.4.2
HTC Sense version: 6.0
Software number: 1.45.401.12
HTC SDK API level: 6.24
Kernel version: 3.4.0-g6544e50 and@ABM102#1 SMP PREEMPT
Baseband version: 1.18.30306251.05G_30.57.306251.00L_F
Build number: 1.45.401.12 CL352881 release-keys

Stock recovery info: Untick "Settings -> Pover -> Fast boot".
Power-off the device.
Hold VolDown hardware key, press and hold power key.
Phone boots into recovery mode:

Recovery mode info in txt format:

M8_DUGL PVT SHIP S-ON
HBOOT-3.18.0.0000
RADIO-1.18.30306251.05G
OpenDSP-v38.2.2-00542-M8974.0311
OS-1.45.401.12
eMMC-boot 2048MB
Jun 10 2014,19:51:53.0

Connect it to the PC, and press power button while "FASTBOOT" selected. It'll change to "FASTBOOT USB", PC will recognise the device and install drivers (if on win). Execute fastboot to get phone info:

>fastboot getvar all
(bootloader) version: 0.5
(bootloader) version-bootloader: 3.18.0.0000
(bootloader) version-baseband: 1.18.30306251.05G
(bootloader) version-cpld: None
(bootloader) version-microp: None
(bootloader) version-main: 1.45.401.12
(bootloader) version-misc: PVT SHIP S-ON
(bootloader) serialno: ************
(bootloader) imei: ***************
(bootloader) imei2: ***************
(bootloader) meid: 00000000000000
(bootloader) product: m8_dugl
(bootloader) platform: hTCBmsm8974
(bootloader) modelid: 0P6B64000
(bootloader) cidnum: HTC__032
(bootloader) battery-status: good
(bootloader) battery-voltage: 0mV
(bootloader) partition-layout: Generic
(bootloader) security: on
(bootloader) build-mode: SHIP
(bootloader) boot-mode: FASTBOOT
(bootloader) commitno-bootloader: 6b903f73
(bootloader) hbootpreupdate: 11
(bootloader) gencheckpt: 0
all: Done!
finished. total time: 0.027s

According to this, CID/MID of this device are HTC__032/0P6B64000
Then I proceeded with the unlock procedure.
Unlock wipes the device to factory reset.

After the unlock, I rooted device following this guide - tried several recovery images, but only TWRP_Recovery_2.8.2.1_M8_CPTB.img worked (booting recovery image is enough, you don't need to flash it). After playing with rooted phone for some time I finally figured out that for real control over the device I do need S-OFF. At the time of writting the only confirmed way to S-OFF for dual sim vaiant (m8_dugl) was SunShine. My last Chinese purchase did not pass airline security, I got refunded to PayPall and had some spare money. So, investing into S-OFF and supporting the software was not a bad idea.

After enabling S-OFF I've updated OTA towards latest 3.33.401.6. Finally ended up with m8dugl_3.33.401.6_stock_rooted_deodexed_cptb.zip from here, featuring:

  • Rooted
  • Busybox
  • init.d support
  • wp_mod for system write support
  • symlink for external storage added for Titanium Backup/legacy app compatibility
  • boot image unsecured

As a good base for further modifications.
3.33.401.6 requires different recovery image, and latest official twrp-2.8.6.0-m8.img worked fine.