fabic974 random (binary) wanderings in the philament empires…

Hi, just to let you know that this “web log” is essentially a place where I leave somewhat (im)personal notes to myself.
EDIT/2017: de retour sur l'île de la Réunion, je recherche actuellement un job ou quelque projet captivant auquel je pourrais contribuer, de préférence dans mes cordes: PHP/Symfony, Node.js/Meteor, voire du C/C++ ou du Rust (!) – cf à propos pour plus de détail.

Decrease the value of -s 1500 :

ping -M do -s 1500 -c 1 vps.fabic.net
PING vps.fabic.net (91.134.136.248) 1500(1528) bytes of data.
ping: local error: Message too long, mtu=1500

--- vps.fabic.net ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Until we get a ping reply :

$ ping -M do -s 1464 -c 1 vps.fabic.net
PING vps.fabic.net (91.134.136.248) 1464(1492) bytes of data.
1472 bytes from 248.ip-91-134-136.eu (91.134.136.248): icmp_seq=1 ttl=50 time=239 ms

--- vps.fabic.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 239.746/239.746/239.746/0.000 ms

Where man ping for -M pmtudisc_option :

Select Path MTU Discovery strategy. pmtudisc_option may be either :

  • do (prohibit fragmentation, even local one) ;
  • want (do PMTU discovery, fragment locally when packet size is large) ;
  • or dont (do not set DF flag).

Found at ArchLinux: OpenVPN.

EOF

20 May 2018 – Discover network MTU with ping

“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” – Martin Fowler, 2008.

04 May 2018 – Martin Fowler any fool can write code...

Basically one may extract the syslog message using jq in this way :

$ journalctl -u ssh -o json-pretty -b | jq '.MESSAGE'

And we may list “failed password authentication” of valid users in this way :

$ journalctl -u ssh -o json -b \
    | jq '.MESSAGE | capture("^Failed password for (?<invalid>invalid user )?(?<user>\\w+)") | [.user, .invalid] | @sh' \
    | sed -e 's/"\(.*\)"/\1/' -e "s/'//g" \
    | sort -u -k1,1
    | grep -v 'invalid user *$'

EOF

08 Apr 2018 – SSH: List failed password-based connections attempts (using jq)

TL;DR: This will query the Github API (v3) endpoint using Curl, and select the SSH/Git URL as well as the fork owner username using jq, that one may process through Sed and a Bash while-read loop for adding Git remotes :

$ curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
    https://api.github.com/repos/ulli-kroll/mt7610u/forks |
      jq '.[] | [.ssh_url, .owner.login] | @sh' |
        sed -e 's/"\(.*\)"/\1/' -e "s/'//g" |
          while read url login ;
          do
            echo "~~> Adding remote '$login' for $url :";
            git remote add "$login" "$url" || (echo "ERROR!") ;
          done

We resort to Sed for removing the wrapping double-quotes; as well as single-quotes.

The most simple query using Curl and basic auth/n :

$ curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
    https://api.github.com/repos/ulli-kroll/mt7610u/forks

See developer.github.com/v3/ about the available authentication mecanisms.

Piping the output through jq, selecting data :

$ curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
    https://api.github.com/repos/ulli-kroll/mt7610u/forks |
      jq '.[] | {name: .full_name, url: .html_url, git: .ssh_url, homepage: .homepage, has_issues: .has_issues, has_wiki: .has_wiki}'

Getting just the list of Git/SSH URLs :

$ curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
    https://api.github.com/repos/ulli-kroll/mt7610u/forks |
      jq '.[] | .ssh_url'

Along with owner login, wrapped into an array :

curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/ulli-kroll/mt7610u/forks |
    jq '.[] | [.ssh_url, .owner.login]'

Filtered with |@sh provides an output that is very close to a table :

curl -s -u fabic -H "Accept: application/vnd.github.v3+json" \
  https://api.github.com/repos/ulli-kroll/mt7610u/forks |
    jq '.[] | [.ssh_url, .owner.login]'

NOTE: There’s also Bash script ok.sh, but it seems not to provide a command for listing forks.

EOF

06 Apr 2018 – Github/API: List forks of a repository + jq.sh

Courtesy: Answer by Victor Klos, circa 2015

function geekiest_method_queries_stun_server() {
  curl -s http://olegh.ftp.sh/public-stun.txt |
    grep -v '^#' | column -t -s: | shuf -n1 |
    ( read server port &&
      echo "Querying STUN server '$server' on port $port" &&
      echo -en '\x00\x01\x00\x08\xc0\x0c\xee\x42\x7c\x20\x25\xa3\x3f\x0f\xa1\x7f\xfd\x7f\x00\x00\x00\x03\x00\x04\x00\x00\x00\x00' |
      nc -u -w 2 "$server" "$port" |
      dd bs=1 count=4 skip=28 2>/dev/null |
      hexdump -e '1/1 "%u."' |
      sed 's/\.$/\n/' )
}

~$ geekiest_method_queries_stun_server

Wrapped up in that script of yours.

EOF

03 Apr 2018 – Query STUN server for your public IP

Dude, u do this for monitoring traffic as you fiddle with Linux’ Netfilter/iptables :

~# watch -d -n1 'echo "--- MANGLE ---\n" ; iptables -t mangle -nvL ; echo "\n--- FILTER ---\n"; iptables -t filter -nvL ; echo "\n--- NAT ---\n"; iptables -t nat -nvL'
03 Apr 2018 – Netfilter/iptables: watch command

Query current XKB keyboard states with :

$ setxkbmap -query

One-liner using setxkbmap :

$ setxkbmap -model tm2030USB \
            -layout us       \
            -variant ,altgr-intl,alt-intl        \
            -option "" -option grp:shifts_toggle \
            -verbose \
  && echo
  && setxkbmap -query

Note: the trailing , on the -variant ,alt-intl ... argument, means "" empty string => use the default variant for the specified layout.

‘tis better to use localectl nowadays, for it will update a conf. file located at /etc/X11/xorg.conf.d/00-keyboard.conf :

$ localectl set-x11-keymap us tm2030USB ,altgr-intl,alt-intl grp:shifts_toggle

List supported keyboard models, layouts, layout variants and options :

$ localectl list-x11-keymap-models
$ localectl list-x11-keymap-layouts
$ localectl list-x11-keymap-variants us
$ localectl list-x11-keymap-options

References :

EOF

09 Mar 2018 – Xorg: quickfix keyboard layout with setxkbmap or localectl

This will list the hard-coded compiler include paths :

clang++ -Wp,-v -x c++ - -fsyntax-only < /dev/null

Or :

clang++ -E -x c++ - -v < /dev/null

Using Sed to select those include path (SO):

$ clang++ -Wp,-v -x c++ - -fsyntax-only < /dev/null 2>&1 |
    sed -e '/^#include <...>/,/^End of search/{ //!b };d'

Where //!b :

@Brendan the instruction //!b reads if the current line is neither one of the lines that match the range, break and therefore print those lines otherwise all other lines are deleted. – potong Feb 17 ‘17 at 1:14

EOF

22 Jan 2018 – Clang/GCC: find out default include path

Yet another onliner for cleaning up untracked files from a checked-out Git tree.

$ git status -uall --porcelain \
    | grep -E '^\?\? +'        \
    | awk '{ print $2 }'       \
    | xargs -rt -d\\n rm -v

Or move to the trash instead with gio trash ... :

$ git status -uall --porcelain \
    | grep -E '^\?\? +'        \
    | awk '{ print $2 }'       \
    | xargs -rt -d\\n gio trash

EOF

11 Jan 2018 – Git: remove untracked files onliner

It is possible to use Bash expressions as arguments to git submodule foreach ... Hence given a project with loads of dependencies checked-out as submodules, updating only a subset of these that reside in, say vendor/extra/, may be achieved with :

cd vendor/extra/ &&
  git submodule foreach \
    '[[ $path =~ ^../ ]] || (time git remote update)'

Enter each submodule that live under the current directory, and checkout branch release_60 that each have (it’s LLVM/Clang btw.) ; upon error we’re dropped into a Bash shell for fixing things :

git submodule foreach       \
  '[[ $path =~ ^../ ]] ||   \
                            \
      (git checkout -b release_60 origin/release_60 || bash)'
04 Jan 2018 – Git submodule foreach selective update

Use Bash substitution ${var+x} for testing whether or not a variable (incl. environment one) is set or not (via).

[ -z ${EDITOR+x} ] && echo "\$EDITOR IS NOT SET" || echo "SET: $EDITOR" ]

Example :

[ -z ${CC+x}  ] && type -p clang >/dev/null
  && CC=clang
  && echo "| \$CC was set to $CC"

Likewise for CXX :

[ -z ${CXX+x} ] && type -p clang++ >/dev/null
  && CXX=clang++
  && echo "| \$CXX was set to $CXX"

EOF

21 Dec 2017 – Bash: Test if variable is set (exists) or not

This one command will attempt connection on port 11371 at pgp.mit.edu :

gpg --keyserver hkp://pgp.mit.edu \
    --recv-keys A4A9406876FCBD3C456770C88C718D3B5072E1F5

Protocol scheme hkp:// is for port 11371 :

grep hkp /etc/services

Tought it might be my firewall, but looking at the output of sudo watch -d -n1 iptables -nvL showed no packets being dropped at all. Didn’t go look at my home router, might be that it filters out traffic (?).

Anyway, found out that one may request keys through port 80 :

gpg --keyserver hkp://pgp.mit.edu:80 \
    --recv-keys A4A9406876FCBD3C456770C88C718D3B5072E1F5

Story: Trying to build mysql-connector-cpp on ArchLinux (AUR) and makepkg -si fails with:

==> Verifying source file signatures with gpg...
    mysql-connector-c++-1.1.9.tar.gz ... FAILED (unknown public key 8C718D3B5072E1F5)
    ==> ERROR: One or more PGP signatures could not be verified!
19 Dec 2017 – MySQL: Couldnt fetch GPG key from pgp.mit.edu port 11371

Build SOCI (github.com/SOCI/soci).

$ git clone git@github.com:SOCI/soci.git vendor/soci &&
    cd vendor/soci/src/ &&
      ( [ -d build ] && rm -r build ) &&
      mkcd build &&
      cmake -G Ninja \
        -DCMAKE_BUILD_TYPE=RelWithDebInfo \
        -DSOCI_STATIC=ON \
        -DWITH_BOOST=ON \
        -DWITH_MYSQL=ON \
          ../src/ &&
      ninja -j2 ;
      echo "RETV=$? " ;
      pwd

EDIT: -_- dropped it, caused some seg. fault that I couldn’t solve.

19 Dec 2017 – Build SOCI C++ database abstraction library

Using awk to sum the output of du (SO):

$ find /usr/include/ -type f \
    -iregex '.+\.\(c\|h\|hh\|cpp\|hpp\|cxx\|hxx\|h\.inc\|s\)$' \
      | xargs -r du -b \
      | awk '{s+=$1} END {print s}'

One-liner, using Bash’s $(( ... )) to compute the size in kilo-bytes:

$ echo "Size of your system header files: $(( $(find /usr/include/ -type f -iregex '.+\.\(c\|h\|hh\|cpp\|hpp\|cxx\|hxx\|h\.inc\|s\)$' | xar
gs -r du -b | awk '{s+=$1} END {print s}') / 1024 )) kBytes."

Size of your system header files: 270115 kBytes.
17 Dec 2017 – Quick compute size of system header files

Bash provides the shell builtin ulimit, and specifically ulimit -v <kBytes> which sets the maximum amount of virtual memory (i.e. heap) that it can allocate, and it so happens that sub-processes will inherit this value as well.

$ ulimit -v $((512*1024))

Quick test :

cat <<EOF | clang++ -x c++ - && ( echo HEY; ulimit -v $((512*1024)) ; ./a.out )
#include <iostream>
using namespace std;

int main() {
  cerr << "HEY";
  unsigned long amount = 0;
  const unsigned increment = 1024*1024;
  while(true) {
    char *buf = new char[ increment ];
    amount += increment;
    cerr << "\n" << amount << "        ";
    cerr << flush;
  }
  return -1;
}
EOF
  • TODO: Find out how to do this by resorting to cgroups in an easy manner.
  • Note that sub-processes will inherit this same max. virtual memory limit, so we cannot restrict the maximum amount of memory of a whole process tree in this way.
  • Story: while doing some Jekyll dev. I ran into {{ site | jsonify }} freeze my whole system by consuming memory infinitely, certainly due to the fact that jsonify does not handle cycles. And I did it again when writing this line -_-.
17 Dec 2017 – Linux limit [sub]process(s) maximum memory

Query S.M.A.R.T. information from all hard-disks :

# for dev in /dev/sd? ; do
    echo -e "\n\n### DEV: $dev ###\n\n" ;
    smartctl -a $dev || break ;
    read -p "-- ^ $dev -- Continue ? --" ;
  done
  • Use smartctl -t short /dev/sdX to initiate a short self-test (long for extented self-tests).

smartd may be configured (/etc/smartd.conf) to schedule automatic run of self-tests, albeit resorting to some arcane configuration, here (S/../.././12|L/../../6/19), where (I guess) ‘S’ stands for Short, and ‘L’ for Long :

# /etc/smartd.conf

# Short self-test everyday between 12:00 - 13:00 ;
# and extended self-test on Saturdays btw. 19:00 20:00 :
DEVICESCAN -a -o on -S on -n standby,q \
  -W 4,35,40                           \
  -s (S/../.././12|L/../../6/19)       \
  -m cadet.fabien+sys@gmail.com
  • Arch wiki: SMART
  • Note that :
    • The first DEVICESCAN ... encountered will stop smartd from reading the rest of its configuration file.
    • Configuration can be specified per disk, ex. /dev/sda -a -W 2,35,40.

    • And that disks may be specified by their UUID instead of their block device path /dev/sdX so that configuration is not affected by e.g. some internal re-wiring of the drives on the motherboard/controller(s), see ls -l /dev/disk/by-uuid/ :

         /dev/disk/by-uuid/09022016-0924-1909-2411-2709cade0da3 -a -o on -S on
14 Dec 2017 – Check all hard-drives SMART info.

ZFS installed from the AUR causes Pacman to fail during dependency resolution (due to the fact that the AUR packages are tied to a specific (strict) Linux kernel version). Bypass the problem by just ignoring the upgrade of the Linux kernel :

# pacman -Syu \
    --ignore spl-linux,spl-linux-headers,zfs-linux,zfs-linux-headers,linux,linux-headers,linux-firmware
  • TODO: Find out the correct way to upgrade the whole system, including ZFS stuff, for ex. DKMS
  • And generally, find out how to “correctly” manage your bunch of installed AUR packages.
  • ZFS @ ArchLinux
14 Dec 2017 – ArchLinux & ZFS: Upgrade system howto

“Instead of the programs I had hoped for, there came only a shuddering blackness and ineffable loneliness; and I saw at last a fearful truth which no one had ever dared to breathe before — the unwhisperable secret of secrets — The fact that this language of stone and stridor is not a sentient perpetuation of Rust as London is of Old London and Paris of Old Paris, but that it is in fact quite unsafe, its sprawling body imperfectly embalmed and infested with queer animate things which have nothing to do with it as it was in compilation. ”
The Rustonomicon ( doc.rust-lang.org/nomicon/ )

lol

12 Dec 2017 – Quote: The Rustonomicon – shuddering blackness & ineffable loneliness. 
~#  mkfs.ext4 -L archlinux -m 1 -U time -v /dev/sda3
  • Note that default filesystem features (-O ...) are read from /etc/mke2fs.conf.
  • -L ... : filesystem label.
  • -m 1 : 1% reserved blocks pct.
  • -U time : generate UUID from date-time.
09 Dec 2017 – Write ISO image to USB stick (dd)

Fast blank the media (optional, possibly superfluous) :

$ time \
    wodim -v speed=1 -sao dev=/dev/sr0 blank=fast

Burn the ISO image file to disk :

$ time \
    wodim -v speed=1 -sao dev=/dev/sr0 \
      archlinux-2017.12.01-x86_64.iso

Verify data by computing a SHA-1 fingerprint. Arg. count=$(( ... )) evaluates the amount of 2048-bytes blocks that the source ISO file have. This is recommanded (as per Arch wiki) because the optical drive may (or may not) read some additional garbage.

$ time \
    dd if=/dev/sr0 \
    bs=2048        \
    count=$(( $(du -b archlinux-2017.12.01-x86_64.iso | awk '{print $1}') / 2048 )) \
      | sha1sum -b
  • -V (capital ‘V’) will display very verbose SCSI layer debugging details.
  • -scanbus or --devices is supposed to find out usable devices, but didn’t work -_- *
  • ArchLinux: Optical_disc_drive
09 Dec 2017 – Burn ISO image to CD / DVD

Yet another ultimate Rsync command line :

time \
  nice \
    rsync -aHAX -yy -z --inplace -vihP --stats        \
        /some/source/dir1 /some/other/source/dir2 ... \
        /target/dir       \
      --delete-delay -n ; \
          echo "RSYNC DONE, RETV=$? `date`"

But usually rsync -aviHAXP will do the job.

08 Dec 2017 – Rsync

Best option for obtaining I/O statistics is iotop :

$ sudo iotop -oP -d3

But Linux dist. does not have it (?), resort to iostats – This will output some I/O statistics (at most one terminal screen) :

$ iostat -cdkx 2 $(($LINES/6))

Output through watch yields slightly strange results :

$ S_COLORS=always watch -c -n1 iostat -cdkx 1 8

In a loop :

$ while true; do iostat -cdkx 2 $(($LINES/6)); clear; done

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30.80    0.32    5.97    0.22    0.00   62.70

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              2.59    1.80    137.88    114.07     0.40     3.09  13.32  63.21    0.82   12.31   0.02    53.30    63.37   0.52   0.23
sdb              0.00    0.08      0.00     33.35     0.00     0.16   4.77  67.12   65.94  370.49   0.03     5.77   428.41   4.80   0.04

TODO: Write down some notes about these numbers (e.g. average queue length).

08 Dec 2017 – iostats

Generate an image with my email address, from the command line :

$ convert -background 'rgba(0,0,0,0)' -fill black \
    -font Source-Code-Pro-for-Powerline -pointsize 12 \
    label:"noreply@examble.com" \
    cadet.fabien_at_gmail.black.png

Likewise for my mobile phone number :

$ convert -background 'rgba(0,0,0,0)' -fill black \
    -font Source-Code-Pro-for-Powerline -pointsize 12 \
    label:"+33 (0) 123 11 22 33" \
    cadet_fabien_mobile_phone_number.black.png

List available fonts :

$ convert -list font | grep Font:

There seems to exist a possibly “better” alternative to ImageMagick, probably a fork of it : GraphicsMagick. But it behaves slightly differently.

$ sudo apt-get install graphicsmagick

$ gm convert -background 'rgba(0,0,0,0)' -fill white \
    -font Source-Code-Pro-for-Powerline -pointsize 12 \
    label:"+33 (0) 123 11 22 33" \
    cadet_fabien_mobile_phone_number.white.png
08 Dec 2017 – ImageMagick gen. text to image file

A data race is a particular type of race condition in which these three behaviors occur [Rust]:

  • Two or more pointers access the same data at the same time.
  • At least one of the pointers is being used to write to the data.
  • There’s no mechanism being used to synchronize access to the data.
06 Dec 2017 – Data race (definition)

Scalability is the ability of an application to serve the increasing number of requests with no compromise in performance.

23 Nov 2017 – Scalability

“Tzu-li and Tzu-ssu were boasting about the size of their latest programs. ‘Two-hundred thousand lines,’ said Tzu-li, ‘not counting comments!’ Tzu-ssu responded, ‘Pssh, mine is almost a million lines already.’ Master Yuan-Ma said, ‘My best program has five hundred lines.’ Hearing this, Tzu-li and Tzu-ssu were enlightened.”
Master Yuan-Ma, The Book of Programming

08 Nov 2017 – Quote: Master Yuan-Ma

“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”
C.A.R. Hoare, 1980 ACM Turing Award Lecture

08 Nov 2017 – Quote: C.A.R. Hoare on software design

“Just as we cannot think of spatial objects at all apart from space, or temporal objects apart from time, so we cannot think of any object apart from the possibility of its connection with other things.”
Ludwig Wittgenstein

30 Sep 2017 – Quote: Thinking in Time & Space

My dad passed away this morning, 11.40 AM; it was a sunny day; we knew this might happen, but didn’t expect this so soon; have a nice trip over there, see you soon.

11 Sep 2017 – My dad passed away...

« 90% des gens sont prêts à croire n’importe quoi, si on leur donne un pourcentage… »
inconnu

06 Aug 2017 – Citation: pourcentages

Livre: L’espace d’un an – Becky Chambers

book cover

Un bouquin de SF bien sympa., ambiance légère et beaucoup d'humour – bon ok, on voyage pas mal cloîtré sur le vaisseau, en compagnie des personnages et dans l’attente affairée de la prochaine escale – ça se lit bien, sans stress, et laisse de bons souvenirs !–

23 Apr 2017 – Livre: L'espace d'un an

« Beaucoup d’entre nous mourront ainsi sans jamais être nés à leur humanité, ayant confiné leurs systèmes associatifs à l’innovation marchande, en couvrant de mots la nudité simpliste de leur inconscient dominateur. »
Henri Laborit

24 Jan 2017 – Citation: Henri Laborit

“And now,
Where must we go…
We who wander this wasteland
In search of better selves ?”
The First History Man (Mad Max Fury Road).

Et à présent…
Où devons-nous aller pour être plus justes ?

Cité à la fin de Mad Max Fury Road – cf Quora pour une interprétation plausible.

27 Jun 2015 – Quote: Mad Max - Fury Road

« Les souvenirs sont plus fidèles que les amis et les amants : ils reviennent nous voir lorsque notre âme grelotte toute seule. »
Ferenc Mora

22 Jun 2012 – Citation: Ferenc Mora - les souvenirs

“It takes a long time to make something complicated simple, but if you do, it will work without problems for a long time.”
F. Andy Seidl

19 Nov 2011 – Quote: Andy Seidl, KISS

« Les hommes inventèrent l’aurevoir, parce qu’ils se savent en quelque manière immortels, tout en s’estimant contingents et éphémères. »
Borges

26 Oct 2011 – Citation: aurevoirs (Borges)

“All I know is that I know nothing.”
Socrates

(Whereas Jon Snow knows nothing, but doesn’t know it -_-).

20 Oct 2011 – Quote: Socrates knows nothing

“An expert is a person who has made all the mistakes that can be made in a very narrow field.”
Niels Bohr

18 Oct 2011 – Quote: experts (Niels Bohr)

“One thing life has taught me: if you are interested, you never have to look for new interests. They come to you. When you are genuinely interested in one thing, it will always lead to something else.”
Eleanor Roosevelt

14 Oct 2011 – Quote: E. Roosevelt - genuine interest

« La normalité n’est que le consensus social de la majorité exerçant une pression aliénante sur les éléments qu’elle n’englobe pas. »
Jay Abeganski

11 Oct 2011 – Citation: Normalité, un consensus social

Some old posts...

Youtube