wiki - User contributions [en]

Processes

2026-10-15T12:35:49Z

Hdridder:

[[category:Linux/Unix]]

;ps -aux
:Full information on all running processes.
<syntaxhighlight lang=text>
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 191424 4420 ? Ss 2019 63:30 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root 2 0.0 0.0 0 0 ? S 2019 0:09 [kthreadd]
root 3 0.0 0.0 0 0 ? S 2019 5:28 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 2019 0:00 [kworker/0:0H]
root 7 0.0 0.0 0 0 ? S 2019 0:05 [migration/0]
root 8 0.0 0.0 0 0 ? S 2019 0:00 [rcu_bh]
root 9 0.1 0.0 0 0 ? S 2019 341:06 [rcu_sched]
root 10 0.0 0.0 0 0 ? S< 2019 0:00 [lru-add-drain]
</syntaxhighlight>

;kill <pid>
:Send the TERM signal (usually 9) to the process with PID-<pid>
;killall <commandname>
:Kill all processes with the name <commandname> this is the name as shown by ps -ef or ps -aux without any options or parameters.

;nohup <cmd> &
:Run <cmd> in the background with no connection to the terminal. Closing the terminal does not stop the command from running. All output is sent do a <code>nohup.out</code> file in the current directory.
;jobs
:Show the processes belonging to yours.

For a command running on your terminal with CRTL-z you pause the process and are returned to the bash prompt.
;bg
:Put the process in the background, it starts running again
;disown
:Disconnect the process from the parent. You can not close the terminal without stopping the process. You cannot see this process with jobs anymore.
;ps -u $USER -f
:Show all processes for your username (-f for full format)

Gpg

2026-10-09T14:02:19Z

Hdridder: Created page with "GNU Privacy Guard [https://gnupg.org/][https://en.wikipedia.org/wiki/GNU_Privacy_Guard] ;gpg --list-keys :Show your keys ;gpg --fetch-keys "<URL to keys>" :Import keys ;curl "<URL to keys>" -o <outputfile> :Download keys to outputfile ;gpg --import <file> :import keys from file ;gpg --verify <signature file> <package file> :Verify if the signature file matches the package file. If the file was changed after the signature file was created someone may have tampered it. :I..."

GNU Privacy Guard [https://gnupg.org/][https://en.wikipedia.org/wiki/GNU_Privacy_Guard]

;gpg --list-keys
:Show your keys
;gpg --fetch-keys "<URL to keys>"
:Import keys
;curl "<URL to keys>" -o <outputfile>
:Download keys to outputfile
;gpg --import <file>
:import keys from file
;gpg --verify <signature file> <package file>
:Verify if the signature file matches the package file. If the file was changed after the signature file was created someone may have tampered it.
:If OK the results has: <code>Good signature from ....</code>

If you are sure a key is from a trustworthy person you can trust it in the gpg software directly or from the command-line.
;gpg --export-ownertrust > trust.txt
:Export the trust database
;gpg --import-ownertrust < trust.txt
:Import into the trust database

MariaDB/MySQL

2026-10-04T13:21:51Z

Hdridder: /* Login to password user without plain text password on commandline */

[https://mariadb.org/ MariaDB] is an open source fork of [http://www.mysql.com/ MySQL]. There are some differences but most are internal, for users the commands and [[SQL]] syntax is exactly the same.

[https://www.phpmyadmin.net/ phpMyAdmin] is a web based GUI for the maintenance of MariaDB and MySQL servers.

=Performance related queries=
;show status like 'Qcache_total_blocks'
:Fragmentation of the query cache (lower is better)
;flush query cache
:Flush the query cache to fix the fragmentation (check if cache sizes need to be changed too)

=User management=
User data is stored in table mysql.user
;alter user username@host identified by 'password'
:Change a users password
;drop user username@host
:Delete a user
;grans all privileges on *.* to 'user'@'host';
:Grant all privileges to all database to a user

=Login to password user without plain text password on commandline=
Put the password in a protected config file.

vi ~/.my.cnf
[client]
password=<password>

chmod 600 ~/.my.cnf

When you want to use a different filename start the mysql-client like
mysql --defaults-file=/home/francis/mysql-opts

NOTE: commands can be stored in the history logging. Make sure the log-file (~/.mysql_history) has very restrictive permissions

=Move users from one server to another=

Starting on the source server execute this script
<syntaxhighlight lang=bash>
PASSWORD=$1
mysql -B -N -uroot -p${PASSWORD} -e "SELECT CONCAT('\'', user,'\'@\'', host, '\'') FROM user WHERE user IN ('username1', 'username2', 'username3')" > users.txt
while read line
do
mysql -B -N -uroot -p${PASSWORD} -e "SHOW GRANTS FOR $line"
done < users.txt > users.sql
sed -i 's/$/;/' users.sql
</syntaxhighlight>

Move the file to the other server and there do:

<code>mysql -u root -p < users.sql</code>

=Set up replication=
With replication all transactions on the master database server are immediately replicated to the slave database server. So you create a hot copy of the database that can be used in case of a failure of the master server or to offload the master server by executing retrieval queries on the slave.

The slave server logs into the master and checks if the bin-log position differs, if so all transactions after the last binlog-position are redone on the slave.

This procedure is for a large part based on a tutorial found on howtoforge.com [https://www.howtoforge.com/mysql_database_replication]

For multi master replication to 1 slave check [https://mariadb.com/kb/en/multi-source-replication/ this].

==On the master==
;Edit the my.cnf so it has below lines.
:The server must be accessible from the slave server and the server-id must be unique in you network
:The binlog-do-db specifies for which database the binlog is written, one line per database. If omitted for all databases binlog is written except those specified by binlog-ingnore-db.
#skip-networking
#bind-address = 127.0.0.1
log-bin = /var/log/mysql/mysql-bin.log
#binlog-do-db=exampledb
binlog-ignore-db = mysql
binlog-ignore-db = information_schema
binlog-ignore-db = performance_schema
binlog-ignore-db = phpmyadmin
server-id=1

;Restart the database server
/etc/init.d/mysql restart

;Create a replication user
mysql -uroot -p
mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY '<some_password>';
mysql> FLUSH PRIVILEGES;

;Make sure nothing is changes in the database during backup
mysql> USE exampledb;
mysql> FLUSH TABLES WITH READ LOCK;
Or, for all databases;
mysql> FLUSH TABLES WITH READ LOCK;
mysql> SET GLOBAL read_only = 1;
;Check status;
mysql> SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.006 | 183 | | |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

;Dump the database:
mysqldump -u root -p<password> --opt exampledb > exampledb.sql

;Unlock the tables
mysql -u root -p<password>
mysql> UNLOCK TABLES;
mysql> quit;
Or, for all databases;
mysql> SET GLOBAL read_only = 0;
mysql> UNLOCK TABLES;

==On the slave server==

;Create the slave database
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CREATE DATABASE exampledb;
quit;
</syntaxhighlight>

;Load the database dump created on the master
<code>mysql -u root -p<password> exampledb < /path/to/exampledb.sql</code>

;Edit the my.cnf so it has these lines
:The server-id must be unique in you network
:The replicate-do-db specifies for which database the binlog on the master is read, one line per database. If omitted for all databases binlog is read.
<syntaxhighlight lang=text>
server-id=2
replicate-do-db=exampledb
</syntaxhighlight>

;Restart the database server
<code>/etc/init.d/mysql restart</code>

In the following statement the values from SHOW MASTER STATUS above must be used.

<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
STOP SLAVE;
CHANGE MASTER TO MASTER_HOST=<masterIP>, MASTER_USER='slave_user', MASTER_PASSWORD='<password>', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=183;
START SLAVE;
quit;
</syntaxhighlight>

====In case of using GTID for replication====
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CHANGE MASTER TO master_host= <masterIP>', master_user='<slave_user>', MASTER_PASSWORD='<password>', master_use_gtid=current_pos;
</syntaxhighlight>

;<code>SELECT @@GLOBAL.gtid_slave_pos;</code>
:Show current GTID position
;<code>SET GLOBAL gtid_slave_pos = ""</code>
:Reset counter, slave databases must be in sync with the master.

=Add a database to replication=

Everything must be working fine and log positions up to date. On the slave Read_Master_Log_Pos is the same as the Exec_Master_Log_Pos and the Read_Master_Log_Pos is the same as the masters Position (in show master status).

Then you can do:
* Slave: Add database to my.cnf (replicate-do-db=<dbname>)
* Slave: <code>stop slave;</code>
* Master: <code>flush tables with read lock;</code>
* Master: dump database
* Master: <code>unlock tables</code>
* Slave: Load the database dump
* Slave: Restart the database server
Then check the slave status <code>show slave status \G</code>

=Repair replication=
==On the slave server==
;Stop replication processes
mysql -u root -p
mysql> stop slave;

==On the master server==
Dump the replicated databases including the replication data.
<code>mysqldump --databases <db1> <db2> --master-data=1 > ../mysql_dump_$(hostname)_$(date +%Y-%m-%d_%H-%M).sql.dump</code>

==On the slave server==
;Import the databases
<code>mysql -uroot -p < <dumpfile></code>
;Restart the slave processes and check the slave status.
When needed update the slave log-file and position as in the masters <code>show slave status</code>
mysql> CHANGE MASTER TO MASTER_LOG_FILE='<filename>', MASTER_LOG_POS=<postition>;
Then:
mysql> start slave;
mysql> show slave status \G

This should not show any errors

=Populate and update timezone information=
Timezone information is required to use functions like CONVERT_TZ. The mysql.time_zone table need to be populated and maintained when the data changes.

On a [[synology]] server following command takes care of this:

<code>/volume1/@appstore/MariaDB10/usr/local/mariadb10/bin/mysql_tzinfo_to_sql /usr/share/zoneinfo |mysql -uroot -p mysql</code>

MariaDB/MySQL

2026-10-04T13:09:15Z

Hdridder: /* Login to password user without plain text password on commandline */

[https://mariadb.org/ MariaDB] is an open source fork of [http://www.mysql.com/ MySQL]. There are some differences but most are internal, for users the commands and [[SQL]] syntax is exactly the same.

[https://www.phpmyadmin.net/ phpMyAdmin] is a web based GUI for the maintenance of MariaDB and MySQL servers.

=Performance related queries=
;show status like 'Qcache_total_blocks'
:Fragmentation of the query cache (lower is better)
;flush query cache
:Flush the query cache to fix the fragmentation (check if cache sizes need to be changed too)

=User management=
User data is stored in table mysql.user
;alter user username@host identified by 'password'
:Change a users password
;drop user username@host
:Delete a user
;grans all privileges on *.* to 'user'@'host';
:Grant all privileges to all database to a user

=Login to password user without plain text password on commandline=
Put the password in a protected config file.

vi ~/.my.conf
[client]
password=<password>

chmod 600 ~/.my.cnf

When you want to use a different filename start the mysql-client like
mysql --defaults-file=/home/francis/mysql-opts

NOTE: commands can be stored in the history logging. Make sure the log-file (~/.mysql_history) has very restrictive permissions

=Move users from one server to another=

Starting on the source server execute this script
<syntaxhighlight lang=bash>
PASSWORD=$1
mysql -B -N -uroot -p${PASSWORD} -e "SELECT CONCAT('\'', user,'\'@\'', host, '\'') FROM user WHERE user IN ('username1', 'username2', 'username3')" > users.txt
while read line
do
mysql -B -N -uroot -p${PASSWORD} -e "SHOW GRANTS FOR $line"
done < users.txt > users.sql
sed -i 's/$/;/' users.sql
</syntaxhighlight>

Move the file to the other server and there do:

<code>mysql -u root -p < users.sql</code>

=Set up replication=
With replication all transactions on the master database server are immediately replicated to the slave database server. So you create a hot copy of the database that can be used in case of a failure of the master server or to offload the master server by executing retrieval queries on the slave.

The slave server logs into the master and checks if the bin-log position differs, if so all transactions after the last binlog-position are redone on the slave.

This procedure is for a large part based on a tutorial found on howtoforge.com [https://www.howtoforge.com/mysql_database_replication]

For multi master replication to 1 slave check [https://mariadb.com/kb/en/multi-source-replication/ this].

==On the master==
;Edit the my.cnf so it has below lines.
:The server must be accessible from the slave server and the server-id must be unique in you network
:The binlog-do-db specifies for which database the binlog is written, one line per database. If omitted for all databases binlog is written except those specified by binlog-ingnore-db.
#skip-networking
#bind-address = 127.0.0.1
log-bin = /var/log/mysql/mysql-bin.log
#binlog-do-db=exampledb
binlog-ignore-db = mysql
binlog-ignore-db = information_schema
binlog-ignore-db = performance_schema
binlog-ignore-db = phpmyadmin
server-id=1

;Restart the database server
/etc/init.d/mysql restart

;Create a replication user
mysql -uroot -p
mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY '<some_password>';
mysql> FLUSH PRIVILEGES;

;Make sure nothing is changes in the database during backup
mysql> USE exampledb;
mysql> FLUSH TABLES WITH READ LOCK;
Or, for all databases;
mysql> FLUSH TABLES WITH READ LOCK;
mysql> SET GLOBAL read_only = 1;
;Check status;
mysql> SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.006 | 183 | | |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

;Dump the database:
mysqldump -u root -p<password> --opt exampledb > exampledb.sql

;Unlock the tables
mysql -u root -p<password>
mysql> UNLOCK TABLES;
mysql> quit;
Or, for all databases;
mysql> SET GLOBAL read_only = 0;
mysql> UNLOCK TABLES;

==On the slave server==

;Create the slave database
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CREATE DATABASE exampledb;
quit;
</syntaxhighlight>

;Load the database dump created on the master
<code>mysql -u root -p<password> exampledb < /path/to/exampledb.sql</code>

;Edit the my.cnf so it has these lines
:The server-id must be unique in you network
:The replicate-do-db specifies for which database the binlog on the master is read, one line per database. If omitted for all databases binlog is read.
<syntaxhighlight lang=text>
server-id=2
replicate-do-db=exampledb
</syntaxhighlight>

;Restart the database server
<code>/etc/init.d/mysql restart</code>

In the following statement the values from SHOW MASTER STATUS above must be used.

<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
STOP SLAVE;
CHANGE MASTER TO MASTER_HOST=<masterIP>, MASTER_USER='slave_user', MASTER_PASSWORD='<password>', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=183;
START SLAVE;
quit;
</syntaxhighlight>

====In case of using GTID for replication====
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CHANGE MASTER TO master_host= <masterIP>', master_user='<slave_user>', MASTER_PASSWORD='<password>', master_use_gtid=current_pos;
</syntaxhighlight>

;<code>SELECT @@GLOBAL.gtid_slave_pos;</code>
:Show current GTID position
;<code>SET GLOBAL gtid_slave_pos = ""</code>
:Reset counter, slave databases must be in sync with the master.

=Add a database to replication=

Everything must be working fine and log positions up to date. On the slave Read_Master_Log_Pos is the same as the Exec_Master_Log_Pos and the Read_Master_Log_Pos is the same as the masters Position (in show master status).

Then you can do:
* Slave: Add database to my.cnf (replicate-do-db=<dbname>)
* Slave: <code>stop slave;</code>
* Master: <code>flush tables with read lock;</code>
* Master: dump database
* Master: <code>unlock tables</code>
* Slave: Load the database dump
* Slave: Restart the database server
Then check the slave status <code>show slave status \G</code>

=Repair replication=
==On the slave server==
;Stop replication processes
mysql -u root -p
mysql> stop slave;

==On the master server==
Dump the replicated databases including the replication data.
<code>mysqldump --databases <db1> <db2> --master-data=1 > ../mysql_dump_$(hostname)_$(date +%Y-%m-%d_%H-%M).sql.dump</code>

==On the slave server==
;Import the databases
<code>mysql -uroot -p < <dumpfile></code>
;Restart the slave processes and check the slave status.
When needed update the slave log-file and position as in the masters <code>show slave status</code>
mysql> CHANGE MASTER TO MASTER_LOG_FILE='<filename>', MASTER_LOG_POS=<postition>;
Then:
mysql> start slave;
mysql> show slave status \G

This should not show any errors

=Populate and update timezone information=
Timezone information is required to use functions like CONVERT_TZ. The mysql.time_zone table need to be populated and maintained when the data changes.

On a [[synology]] server following command takes care of this:

<code>/volume1/@appstore/MariaDB10/usr/local/mariadb10/bin/mysql_tzinfo_to_sql /usr/share/zoneinfo |mysql -uroot -p mysql</code>

Help:Wiki

2026-09-25T14:53:10Z

Hdridder: /* Upgrade */

[[Category:Help]]
=Generic=
Wiki formatting options[https://www.mediawiki.org/wiki/Help:Formatting] or check Mediawiki advanced editing[https://www.mediawiki.org/wiki/Help:Advanced_editing] for more.

;<nowiki><syntaxhighlight lang='bash'></nowiki>
;<nowiki></syntaxhighlight></nowiki>
:Block of code formatted for the specified [http://pygments.org/languages/ language]

;<nowiki><nowiki></nowiki>
;<nowiki></nowiki></nowiki>
:Do not apply wiki formatting on this block

;<nowiki>;</nowiki> Definition list term (<nowiki><dt></nowiki>)
:<nowiki>:</nowiki> Definition list definition or just indent(<nowiki><dd></nowiki>)
::<nowiki>::</nowiki> Indent more

;<nowiki>*</nowiki> Bullit list
* Bullit list

;<nowiki>#</nowiki> Numbered list
# Numbered list

;<nowiki>[[category:name]]</nowiki>
:Add this page to category 'name'

;<nowiki>[[wikipage|alt text]]</nowiki>
:Link to page on local wiki. When provided 'alt text' is displayed instead of the page name.

;<nowiki>[[wikipage#Anchor|alt text]]</nowiki>
:Link to anchor on local wiki page. Headings, span and div's are automatic anchors.

;<nowiki><span id=Anchor1></span></nowiki>
;<nowiki><div id=Anchor1></div></nowiki>
;<nowiki>=Heading=</nowiki>
:Create anchor (works for each heading level)

;<nowiki>[URL text]</nowiki>
:Link to URL. When text is not provided the link's consecutive number is displayed.

;<nowiki>''</nowiki>''italics''<nowiki>''</nowiki>
;<nowiki>'''</nowiki>'''bold'''<nowiki>'''</nowiki>
;<nowiki>'''''</nowiki>'''''bold italics'''''<nowiki>'''''</nowiki>
;<nowiki><code></nowiki><code>Fixed width text</code><nowiki></code></nowiki>

=Tables[https://www.mediawiki.org/wiki/Help:Tables]=
{| class="wikitable" style="width:60%"
| style="padding: 10px;"| <big><nowiki>{|</nowiki></big>
| style="padding: 5px;"|  '''table start''', ''required''
|-
| style="padding: 10px;"| <big><nowiki>|+</nowiki></big>
| style="padding: 5px;" |  table '''caption''', ''optional;'' only between '''table start''' and '''table row'''
|-
| style="padding: 10px;"| <big><nowiki>|-</nowiki></big>
| style="padding: 5px;" |  '''table row''', ''optional on first row''—wiki engine assumes the first row
|-
| style="padding: 10px;"| <big><nowiki>!</nowiki></big>
| style="padding: 5px;" |  '''table header''' cell, ''optional.'' Consecutive '''table header''' cells may be added on same line separated by double marks (<code>!!</code>) or start on new lines, each with its own single mark (<code>!</code>).
|-
| style="padding: 10px;"| <big><nowiki>|</nowiki></big>
| style="padding: 5px;" |  '''table data''' cell, ''optional''.> Consecutive '''table data''' cells may be added on same line separated by double marks (<code><nowiki>||</nowiki></code>) or start on new lines, each with its own single mark (<code><nowiki>|</nowiki></code>).
|-
| style="padding: 10px;"| <big><nowiki>|}</nowiki></big>
| style="padding: 5px;" |  '''table end''', ''required''
|}

Example:

<nowiki>{| class="wikitable" style="width:60%"</nowiki><br/>
<nowiki>|+ Caption</nowiki><br/>
<nowiki>! style="width:30%"| Header</nowiki><br/>
<nowiki>! style="width:70%"| Header</nowiki><br/>
<nowiki>|-</nowiki><br/>
<nowiki>| Content</nowiki><br/>
<nowiki>| Content</nowiki><br/>
<nowiki>|}</nowiki><br/>

{| class="wikitable" style="width:60%"
|+ Caption
!Header
!Header
|-
| Content
| Content
|}

=Formula's=
For formula rendering by the Math extension check these [https://meta.wikimedia.org/wiki/Help:Displaying_a_formula examples].

Basic <code><nowiki><math>formula</math></nowiki></code>

=Embedded file (images)=
<nowiki>[[file:RadioSpectrum.jpeg|400x1107px]]</nowiki>. See the [https://www.mediawiki.org/wiki/Help:Images MediaWiki documentation].

=References=
After activating the [https://www.mediawiki.org/wiki/Extension:Cite Cite extentsion] in LocalSettings.php (wfLoadExtension( 'Cite' );) you can put <nowiki><ref>This is the reference</ref></nowiki> in your text. At the end of the page you can put <nowiki>{{References}} or {{reflist}}</nowiki> to have the references displayed.

More on how to use references can be found with wikipedia [https://en.wikipedia.org/wiki/Help:Referencing_for_beginners][https://nl.wikipedia.org/wiki/Help:Referenties_en_voetnoten]

=Add icons to the footer=
Add following to LocalSettings.php
<syntaxhighlight lang=php>
$wgFooterIcons['poweredby']['ipv6ready'] = [
"src" => "https://ipv6-test.com/button-ipv6-small.png",
"url" => "https://ipv6-test.com/validate.php?url=$wgServer",
"alt" => "ipv6 ready",
"height" => "31",
"width" => "88",
];
</syntaxhighlight>
src can be a local file too. More information on the [https://www.mediawiki.org/wiki/Manual:$wgFooterIcons Mediawiki site].

=Sidebar editing=
;Edit the sidebar (not the Tools section there).
:Like editing a wikipage, goto //yourwikiurl/index.php?title=MediaWiki:Sidebar
:Syntax is a bit different, e.g. no <nowiki>[[]]</nowiki> for links
:The preview has little to do with how the sidebar will look. You have to save to see the result
:More info on [https://www.mediawiki.org/wiki/Manual:Interface/Sidebar the MediaWiki manual]

=Redirect=
Redirect pages verwijzen direct door naar een andere pagina. Zij bestaan uitsluitend uit:

<code>#REDIRECT <nowiki>[[pagina]]</nowiki></code>

=Delete page history=
20180817 BEWARE, yet untested
# Prodecure found here [http://www.hungryfools.com/2007/04/how-to-clean-up-change-history-of.html] that just invokes in the maintenance subdirectory <code># php deleteOldRevisions.php --delete</code>
# A more complex [http://www.bigsoft.co.uk/blog/2011/08/03/permanently-deleting-mediawiki-pages-and-revisions suggestion]
# [https://en.wikibooks.org/wiki/MediaWiki_Administrator%27s_Handbook/Page_Deletion#Restoring_some_or_all_deleted_revisions This page] advises to go to Special:Nuke, which does not exist on my wiki.

=Installations and upgrades=

==Upgrade==
* Make sure you have a backup of the current files and database.
* Download and extract mediawiki and your extenstions tar-files over the current.
* Check if LocalSettings.php, your ./images directory are not overwritten
* Make sure ./cache is empty
* Run <code>php ./maintenantce/updade.php</code>
* Link your web-root to the new release
* Rename the old directory to xxx_old to make sure nothing is fetched from any cache (I have to do this despite removing browser history and nginx restarts)

==CirrusSearch==
Search extension that supports more complex search phrases.

For installation on mediawiki 1.35:
* Install java8 and [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/getting-started-install.html#successfully-running-node elasticsearch 6.8.23]
** <code>apt-get install openjdk-8-jre-headless</code>
** <code>echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list</code>
**<code>sudo apt-get update && sudo apt-get install elasticsearch</code>
* Install the Elastica and CirrusSearch (use the same version as your MediaWiki version) extensions [https://www.mediawiki.org/wiki/Extension:CirrusSearch]
* Follow instructions in ./extentions/CirrusSearch/README
** Add this to LocalSettings.php
wfLoadExtension( 'Elastica' );
wfLoadExtension( 'CirrusSearch' );
$wgDisableSearchUpdate = true;
:*<code>php7.4 ./extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php</code>
:*Remove <code>$wgDisableSearchUpdate = true</code> from LocalSettings.php
:*<code>php7.4 ./extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip</code>
:*<code>php7.4 ./extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse</code>
:*Edit LocalSettings.php to have <code>$wgSearchType = 'CirrusSearch';</code>

Check [[Special:Version]] to see if all is installed

To verify CirrusSearch is used add '&cirrusDumpQuery' to a search URL. A json-string will be shown.

See the [[elasticsearch]] page for more

PHP

2026-09-25T13:59:47Z

Hdridder: /* Upgrade */

[[Category:PHP]]
[[category:Webdesign]]
=Basics=
Script language for dynamic web-pages. The syntax and concepts used are similar to [[Perl]]

;Limited number of loops
<syntaxhighlight lang=php>
for ($i = 0; $i <= 6; $i++) {
echo "Loop number: $i ";
}
</syntaxhighlight>

;escapeshellcmd(shellcommand)
:Return the shellcommand with everything that might confuse the shell escaped.
;exec(<cmd>,<outputarray>,<returnvar>)
:Executed <cmd> in the shell, append the command output to he array <outputarray> and set <returnvar> to the returnvalue of the command.
:Also the last line of the command output is returned
;shell_exec(<cmd>)
:Execute <cmd> (output from excapedshellcmd) in the shell, return the command output.

;<nowiki>date("Y-m-d h:i:s")</nowiki>
:Current timestamp (YYYY-mm-dd HH:MM:SS).
:Format "U" is [https://duckduckgo.com/?q=unixtime unixtime] (seconds since epoch)
;sprintf("%7.2f", $float )
:Format a number (7 digits, 2 decimals)
;number_format($number, 2, '.', ',')
:Format $number with 2 decimals, . as decimal separator and , as thousand separator.

;strpos($string, $search,$start)
:Return position of $search exists in $string. When it is not in, return FALSE. Start searching at position $start (optional). Simple test on strpos - <code>if (strpos(x,y)) {}</code> - does not differ between position 0 and FALSE. Test on position 2 or count from the end by setting $start to a negative value (possible since PHP7.1).
;str_replace($old, $new, $string)
:Return $string with $old replaced by $new
;trim($string, 'abc')
:Remove characters a, b and c from the start and the end of $string. If no characters are specified all whitespace will be removed.

;round($float,<precision>)
:Round $float to <precision> decimals (0 is default)

=Regular expressions=
PHP is Perl-like, patterns need delimiters (default is /<pattern>/ )
;preg_match($pattern, $string)
:Return 1 if $pattern exists in $string
;preg_split('/\s*\=+\s*/',$string
:Split $string into an array on a pattern (in this example on white-spaces and equality signs.
;preg_replace('/#.*$/','',$string);
:Replace matches with the pattern in $string (in this example to remove comments from a line)

=Files=
Almost all you need to know about files. For writing open with 'w' and fwrite($fh,$line)
<syntaxhighlight lang=php>
$fh = fopen($filename, 'r') or die('Unable to open '.$file);
while(!feof($fh)) {
$line = fgets($fh);
}
fclose($fh)
</syntaxhighlight>

Read directory
<syntaxhighlight lang=php>
$dir = "./";

// Open a directory, and read its contents
if (is_dir($dir)){
if ($dh = opendir($dir)){
while (($file = readdir($dh))){
echo "filename:" . $file . "\n";
}
closedir($dh);
}
}
</syntaxhighlight>

=Functions=

All variables are local to the function. You have to pass everything it needs, or declare variables as GLOBAL

<syntaxhighlight lang=php>

function func1($arg1, $arg2)
{
GLOBAL $vardeclaredoutside
<codeblock>
return $value;
}
</syntaxhighlight>

Only one value can be returned but you can return arrays.

<syntaxhighlight lang=php>

function func1()
{
<codeblock>
return array ($value1, $value2);
}

list($val1,$val2) = func1();
</syntaxhighlight>

=Arrays=

;arr = []
:(Create) empty array

;arr1 = [val1, val2, val3]
;arr1 = array(val1, val2, val3)
:Create an indexed array. The keys are the range 0..arraylength.

;array_push($arr1,val1,val2)
:Add elements val1 and val2 to the end of $arr1

;$arr1 = array(key1=>val1, key2=>val2, key3=>val3)
;$arr2[key1] = val1
:Create an associative array (like a [[Perl#Hashes]] or a [[Python:DataTypes#Dictionary_or_dict|Python dictionay]])

;count($arr1)
:Return the number of elements (length) of an array

;end($arr1)
:Return the value of the last element in $arr1 and set the pointer to it.

;join(',',$arr1)
:Join elements in arr1 with ',' as separator.

;foreach($arr1 as $value) { codeblock }
:Loop over an indexed array

;foreach($arr1 as $key => $value) { codeblock }
:Loop over an associative array

;unset($arr1[key])
:Remove the key element from $array. Numerical keys will not be reindexed.

=Install=
On debian (and its derivates)
;apt-get install php-fpm php-mysql
:Install php7 and mysql support
:Fix php security by editing <code>/etc/php/7.0/fpm/php.ini</code>
:Uncomment line with cgi.fix_pathinfo and set it to 0

==php repository==
<syntaxhighlight lang=bash>
apt -y install lsb-release apt-transport-https ca-certificates
wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update
sudo apt -y install php7.4
php -v
</syntaxhighlight>

==Upgrade==
Nice overview of how to upgrade can be found on [https://php.watch/articles/php-8-5-installation-upgrade-guide-debian-ubuntu php-watch]

Installing multiple package can easily be done like <code>install php8.3-{fpm,curl,mbstring,mysql,xml}</code>

Unix time

2026-09-18T11:08:23Z

Hdridder:

[[Category: Linux/Unix]]
The number of seconds since the [https://en.wikipedia.org/wiki/Unix_time Unix Epoch] (1-1-1970).

The are online [https://www.epochconverter.com/ epoch converters]

Originally unix time was stored as a 32-bits integer which limits the time to reaching 19 January 2038 03:14:07 UTC [https://en.wikipedia.org/wiki/Year_2038_problem]. Modern computers use a 64-bit field that expands the maximum time far beyond the expected lifetime of the universe.

;date +%s
:Return the current unix time

{| class="wikitable" style="width:20%"
|+ Good to know these seconds
! Timeframe
! Seconds
|-
| Hour
| 3600
|-
| Day
| 86400
|-
| Week
| 604800
|-
| Year
| 31536000
|}

Mail

2026-09-17T07:14:16Z

Hdridder:

[[Category:Linux/Unix]]
[[Category:python]]
=Sending mail with sendmail=
;<code>sendmail -t < <messagefile></code>
:Send the email as in <messagefile>
:The email headers are in <messagefile>. That should look like:
<syntaxhighlight lang=text>
From: me@mydomain.tld
To: user@domain.tld
Subject: Subject of your mail
email text
</syntaxhighlight>

;<code>mailx -s Subject -a <attachmentfile> email@domain.tld</code>
:Send email with attachment after entering optional text and an EOT (CTRL-D).

=Sending mail with python=

<syntaxhighlight lang=python>
#!/usr/bin/env python3

def usage():
print ("Usage: "+__file__)
print("Send email, include what you need in your program")
version = '0.1, 19700101, Initial release'
print("Version: "+version)
sys.exit(1)

def sendmail(mailto,mailfrom = 'NN',mailsubject = '',mailtext = '',attachments = ''):
import smtplib
from email import encoders
from email.message import Message
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import formatdate

outer = MIMEMultipart()
outer['Subject'] = mailsubject
outer['To'] = ", ".join(mailto)
outer['From'] = mailfrom
outer['Date'] = formatdate(localtime=True)
msg = MIMEText(mailtext,'plain')
outer.attach(msg)
if attachments != '':
for attachment in attachments:
filename = attachment.split('/')[-1]
ctype = 'application/octet-stream'
maintype, subtype = ctype.split('/', 1)
fp = open(attachment, 'rb')
msg = MIMEBase(maintype, subtype)
msg.set_payload(fp.read())
fp.close()
# Encode the payload using Base64
encoders.encode_base64(msg)
# Set the filename parameter
msg.add_header('Content-Disposition', 'attachment', filename=filename)
outer.attach(msg)
composed = outer.as_string()
print(composed)
s = smtplib.SMTP('localhost')
print('Sending mail')
s.sendmail(mailfrom, mailto , composed)
s.quit()
return
</syntaxhighlight>

For Python 2 it was
<syntaxhighlight lang=python>
#!/usr/bin/env python2

def usage():
print ("Usage: "+__file__)
print("Send email, include what you need in your program")
version = '0.1, 19700101, Initial release'
print("Version: "+version)
sys.exit(1)

# USER CONFIGURABLE ITEMS

maxmailsize = 10000000

# END USER CONFIGURABLE ITEMS, DO NOT CHANGE ANYTHING BELOW THIS LINE

def main():
attachement1 = filepath
attachement2 = filepath

mailto = ['you@domain.tld']
mailcc = ['someoneelse@anotherdomain.tld','andmore@domain.tld']
mailbcc = ['secret@domain.tld'] # Only include this in the final sendmail call, not in the headers.
mailfrom = 'me@domain.tld' # Some systems do not like this, just remove it from the sendmail call
mailsubject = 'What this is about'
mailtext = "Hi,\n\nI'd like to send you an email.\n\nBest Regards\n\nMe."
sendmail(mailto, mailcc=mailcc, mailbcc=mailbcc, mailfrom=mailfrom, mailsubject=mailsubject, mailtext=mailtext, attachments=[attachment1,attachment2])
return

def sendmail(mailto,mailcc = '',mailbcc = '',mailfrom = 'Sender',mailsubject = '',mailtext = '',attachments = ''):
import smtplib
from email import encoders
from email.message import Message
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

outer = MIMEMultipart()
outer['Subject'] = mailsubject
outer['To'] = ",".join(mailto)
outer['CC'] = ",".join(mailcc)
outer['From'] = mailfrom
msg = MIMEText(mailtext,'plain')
outer.attach(msg)
if attachments != '':
attachtotsize = 0
for attachment in attachments:
attachsize = os.stat(attachment).st_size
if attachtotsize + attachsize > maxmailsize:
mailtext += attachment+" makes the message bigger than "+maxmailsize+", it is not attached.\n"
else:
attachtotsize += attachsize
filename = attachment.split('/')[-1]
ctype = 'application/octet-stream'
maintype, subtype = ctype.split('/', 1)
fp = open(attachment, 'rb')
msg = MIMEBase(maintype, subtype)
msg.set_payload(fp.read())
fp.close()
# Encode the payload using Base64
encoders.encode_base64(msg)
# Set the filename parameter
msg.add_header('Content-Disposition', 'attachment', filename=filename)
outer.attach(msg)
composed = outer.as_string()
s = smtplib.SMTP('localhost')
s.sendmail(mailfrom, mailto+list(mailcc)+list(mailbcc) , composed)
s.quit()
return

main()
</syntaxhighlight>

=Reading mail with mailx=
The web knows more[https://www.computerhope.com/unix/umailx.htm].
;mailx
:Start mailx client and show 1 pages or messages. The Mailx prompt is a '&'
:The current message is marked with a'>' at the start of the line.

;& quit
:Submit the changes you made and exit the client

;& exit
:Exit the client without submitting the changes

;& <RET>
:Read the current message

;& d
:Delete the current message
;&d<id1> <id3>
:Delete the messages with id <id1> and <id2>
;&d<id1>-<id3>
:Delete the messages with id <id1> up to and including <id2>

;& h
:Show a page of headers
;& z
:Show next page of headers
;& z$
:Show last page of headers

;& +
:Go to next message
;& -
:Go to previous message

PHP

2026-09-16T20:07:57Z

Hdridder:

Git

2026-09-01T06:58:02Z

Hdridder: /* Workflow making changes */

=Using Git=

This [https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow Atlassian workflow] looks like a good way to work with git.

Here is a basic overview of how you can work with git@bitbucket:

Initialize git on your computer:
<syntaxhighlight lang=bash>
git config --global user.name "<your name>"
git config --global user.email "<your emailaddress>"
</syntaxhighlight>

* Add your public key in your profile - 'Personal Settings' - 'SSH keys'

==Fresh project (nothing locally yet)==
* Create a "repository" (project) with a git hosting tool (like Bitbucket)
* <code>git clone <Look under the Clone Button</code> Copy (or clone) the repository to your local machine
* <code>git remote set-url --push origin <Look under the Clone Button></code> Set the push URL
* <code>git add <filenname></code> Add a file to your local repository
* <code>git commit -m 'Reason for commit'</code> ]
* <code>git push</code> Send your changes to your master branch

==For a project you have files for already==

* Create a "repository" (project) with a git hosting tool (like Bitbucket)
* <code>git init</code> (initialize the local repository)
* <code>git pull <Repository URL(for bitbucket look under the Clone Button)></code> master
* <code>git add .</code> (Add all file to the local repository)
* <code>git commit -m 'Initial upload'</code>
* <code>git remote add origin <Look under the Clone Button></code> (Connect local to remote repository)
* <code>git push -u origin --all</code> (push all committed file to remote.

==Other things you can do==
* Create a "branch" (version), make a change, commit the change
* Open a "pull request" (propose changes to the master branch)
* "Merge" your branch to the master branch

=Workflow making changes=
;git pull
:Update from remote
;git checkout -b <branchname>
:Create a new branch
;git status
:Check if something changed
;git add --all
:Add all changed files to the repository
;git restore <file>
:Restore a file to it's last committed stage
;git commit -m "Message"
:Commit your changes
;git push
:Upload your repository
When you are ready with your change
;git checkout master
:Go to the master branch
;git pull
:Update from remote, fix conflicts
;git merge <branchname>
:Update master with your changes, fix conflicts
;git push
:Update the remote repository

=Other commands=
Check the [https://www.atlassian.com/git/tutorials/atlassian-git-cheatsheet cheatsheet] for more.
;<code>git branch -d <brachname></code>
:Delete a branch locally only
;<code>git config --local -l</code>
:List your configuration for this repository (in the current directory).
;<code>git remote -v</code>
:Show 'remote' configuration.
;<code>git status -v</code>
:Show all difference between the current directory and the local repository.
;<code>git add/rm <path></code>
:Add or remove files from the list of files to be committed (local files will be removed too).
;<code>git commit -m 'Reason for commit'</code>
:Commit changes (add/rm) to the local repository.
;<code>git commit -a</code>
:Commit all local changes to the local repository.
;<code>git branch <branchname></code>
:Create a new branche
;<code>git checkout <branchname></code>
:Make <branchename> current
;<code>git checkout -b <branchname></code>
:Make <branchename> and make it current (combine above 2 commands)
;<code>git reset HEAD <path></code>
:Reset commit status for <path>
;<code>git fetch</code>
:Fetch all changes made remote
;<code>git merge <branch></code>
:Merge <branch> into the current branch
;<code>git branch --all</code>
:Show all branches in this repository
;<code>git push</code>
:Send the repository remote
;<code>git pull</code>
:Fetch all changes made remote and merge to current branch
;<code>git log</code>
:Show all commits

=Conflict resolution=
Git merge tries to merge files from 2 branches when they differ. If that is not possible (e.g. as both files have changes in the same lines) the file will mark the conflicting parts.

<<<<<<< HEAD
Lines in the current branch
=======
Lines in the merged branch
>>>>>>> <name of the merged branch>

The ======= is the center of the conflict. Check which lines you want to be in and remove everything else (including the conflict markerd).

Pandas

2026-08-04T14:22:28Z

Hdridder: /* Select data */

[[Category:Python]]
Check the [https://pandas.pydata.org/pandas-docs/stable/10min.html 10 minutes to Pandas] too or [https://pythonexamples.org/pandas-examples/ pythonexamples].
;import pandas as pd
:Import the library, we assume this was done on this page

=DataFrame=
Object for tabular data (that is e.g. obtained by read_html).

Each column is a [[Pandas#Series]]

;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}])</nowiki>
:Create a simple dataframe having a range index, 2 rows and 2 columns
;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}],[row1,row2])</nowiki>
:Create a dataframe having a list index. The size of index must match the number of rows.

Single columns can be addressed by there name like in <code>df.columnname.unique()</code> to get unique values in a column.

Single or multiple columns can be addressed as list like in <code>df.set_index([column1,column2])</code> to set an index over 2 columns.

;df.copy
:Make a copy of df ( = will point to the same dataframe, like with [[Python:DataTypes#Dictionary or dict|dictionaries]])

;df.join(table2)
:Like SQL-join merge on index
;df.merge(table2,on=<column>)
:Like SQL-join on <column> = df.index
;dfall.merge(table2, on = <column>, how='left', indicator=True)
:Return df joined with table2. Add column _merge with value 'left_only' to records that only exist in df.
;df = dfall[dfall['_merge'] == 'left_only']
:Return dataframe with all records not in table2
;df = pd.concat(dflist)
:Concatenate all dataframes in dflist

==Information==

;df.info()
:Information on dataframe (Index, size, datatypes per column)
;df.size
:Return the number of cells in a dataframe
;df.shape
:Return the number of rows and columns in a dataframe as [[Python:DataTypes#Tuple|tuple]].
;df.describe()
:Return various attributes for numeric columns in the dataframe. This may indicate if there are outlyers.
;df.head(x)
;df.tail(x)
:Return first/last x data rows of df (5 is the default value for x).

;df.index
:Return the table index (class = pandas.core.indexes.base.Index)

;df.columns
:Return the column headers (class = pandas.core.indexes.base.Index)
;df.dtypes
:Return the data-type of the columns
;df.columnname.dtype
:Return the data-type of column columnname

==Modify==

;<nowiki>df.rename(columns={oldname:newname}, inplace=True)</nowiki>
:Redefine the column headers (modifies the dataframe itself, nothing is returned)

;df.set_index(['col1','col2'])
:Return the dataframe with a new index composed of 2 columns
;df.reset_index()
:Return the dataframe with a default index (range)
;df.sort_index(ascending=False)
:Sort the dataframe on the index. 'True' is the default if 'acsending' is not supplied.
;df.index = df.index.map(str)
:For a data-type on the index

;df.drop(index)
;df.drop([indexes])
;df.drop(range(0,3))
:Remove row(s) from a dataframe.
;df.drop(columns=[listofcolumnstodrop]
:Remove columns from a dataframe.

;df.dropna(thresh=x)
:Return only rows that have at least x columns with a none-NA value. If x < 2 the threshold can be dropped.

;df.loc[<index>,['field1','field2']] = [value1,value2]
:Change values for index <index> in a dataframe
;df.fillna(<value>)
;df[[<column1>,<column2>]] = df[[<column1>,<column2>].fillna(<value>)
:Return dataframe with all NA-values (in the selected columns) replaced by <value>

;table.agg(newname=('columname', np.max))
:Add newname to df with the max of columnname. This named aggregations is only supported from version v0.25

;df.transform(<function>)
:Apply a function to an existing column or the entire dataframe.

;df.assign(newcolumn = <expression>)
:Add column newcolumn to the dataframe
:Expression can be a function like "lambda d: d['column'] * <something>"

;df.apply(<function>)
:Apply a function to the dataframe. By default every element of the table.
;<nowiki>df.apply(lambda x: <function>(x) if x.name in [columnname] else x)</nowiki>
:Apply a function to specified colums
;<nowiki>df[columname] = df[columname].apply(lambda x: <function>(x))</nowiki>
:Change a column in the dataframe by applying <function>

;df.replace(<pattern>,<newvalue>,regex=True)
:Replace <pattern> with <newvalue> in the entire dataframe

;df[columna].mask(df[columna] == origvalue , newvalue, inplace=True)
:Change all cells in columa that have origvalue into newvalue (inplace, so in the dataframe itself).

==Select data==
Use .loc, else colomn-names will be considered too (if I understand this[https://stackoverflow.com/questions/38886080/python-pandas-series-why-use-loc] correctly).

;WARNING!!
:at least with .loc if only 1 item is found is it returned as string(/float?) else as dataframe/series.

;df.[[iloc]][<slice>]
:Return the rows indicated by <slice> as pandas.Series
;<nowiki>df.iloc[:,0:3]</nowiki>
;<nowiki>df[[col0,col1,col2]]</nowiki>
:Return 3 columns from the dataframe
;df.filter(regex=<regex>,axis='columns')
:Return all columns which name matches <regexp>. (axis=1)

;df.loc[<index>]
:Select the row(s) with index <index>
;df.loc[<indexname>].<columnname>
;df.loc[<indexname>].[<columnname>]
:Select only 1 column from a row as pandas series. You may convert that to a list or set immediately by using <code>list(df.loc[<indexname>].<columnname>)</code>
;df.loc[0][0]
:Select a cell in a dataframe with no index or header
;<nowiki>df.loc[lambda d: d[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1].notnull()]</nowiki>
:Select all rows where column1 equals <value> or do not have a null-value.
;<nowiki>df.loc[(df[column1] == <value>) | (df[colomn1] == <value2>)]</nowiki>
:Combine conditions with '|' (or) or '&' (and). Use () on each condition!!
;<nowiki>df.loc[datetime.today()-pd.Timedelta(days=7) : datetime.today()]</nowiki>
:Select all rows where the (datetime)-index is between 7 days ago and today.
:NOTE!! The index must be sorted in ascending order[https://stackoverflow.com/q/67298584/2213323].
;df.last('7D')
:Select the last 7 days in the dataframe. For this als the datetime index must be sorted in ascending order

See also the [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.contains.html#pandas.Series.str.contains pandas series pages]
;df[df['fieldname'].str.contains('regexp',regex=True)]
:Select all rows in df where fieldname contens matches regexp. The field must have a string datetype. If you are looking for a simple string you can leave out the regex=True part.
;df[df['fieldname'].gt(value)]
:Select all rows in df where fieldname is bigger than value. The field must have a numeric datetype

;df.filter(regex=<regex>,axis='index')
;df.filter(regex=<regex>,axis='index').<columnname>
;df.filter(regex=<regex>,axis='index').index
:Return all rows for which in index matches <regexp> or get only the column of the matched indexes. (axis=0 ) or the indexname(s).
:The .index returns the matching indexes as a [[Python:DataTypes#List|list]].

;df.sort_values(<columnname>)
;df.sort_values([<column1>,<colunm2>],ascending=(True,False))
:Return the dataframe sorted on the values in the columns. The second form sorts on column1 first and then on column2, column1 ascending, column2 descending
;df.groupby([column1,column2])
:Return a DataFrameGroupBy object, this is not a dataframe
;df.collname.unique()
:Return a [[Numpy]] array with all distinct values in column 'collname'

;Get all rows from df1 that have an index that exists in df2
:df = df1[df1.index.isin(df2.index)]
;Negate like
:df = df1[~df1.index.isin(df2.index)]

;Or get all intersectioned indices and select by loc
:df = df1.loc[df.index.intersection(df2.index)]

;Better for readibility is to create the indexes first and than filter
<syntaxhighlight lang=python>
commen_idx = df1.index.intersection(df2.index)
df1_only_idx = df1.index.difference(df2.index)
newdf = df1.loc(df1_only_idx)
</syntaxhighlight>

==Use data==
;Print 2 attributes for each row and check for no value
:pd.isna(value) works for Null, NaN (Nod a number) and NaT (not a time) too.
<syntaxhighlight lang=python>
for index, row in df.iterrows():
print(row['ColumnA'],row['ColumnB'])
if ps.isna(row['ColumnA']):
print('Found a NaN value')
</syntaxhighlight>
;In multi-column indexes the index is returned as a tuple
<syntaxhighlight lang=python>for (indexA, indexB), row in df.iterrows():</syntaxhighlight>

;Print all values for the first row
<syntaxhighlight lang=python>
for column, row in df.iteritems():
print(column,row[0])
</syntaxhighlight>

;Loop over columns
<syntaxhighlight lang=python>
for column in df:
print(f"Columnname {column}")
</syntaxhighlight>

;df.iloc[0]
:return the values of the first row as pandas series. If there is only 1 column returns the value.

;df.iloc[0][0]
:return the value of the first column in the first row. Columns can also be identified by name

;df[<column>].quantile(q=0.1)
:Return the 10th percentile (10% of the values in <column> is lower than the 10th percentile)

;df[<column].str.extract(r"A regex(p\w+)")
:Return a series (column) of the values selected by [[Regular Expressions]] (p\w+)
;df[<columna>].map(str) + '_' + df[<columnb>].map(str)
:Return a series (column) with the columns concatenated. If not mapped to str an error will be thrown.
;<nowiki>df[<column>].map(str).apply(lambda x: x[-1])</nowiki>
:Return a series (column) with the last character of the <column>

=Series=
Pandas Series [https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html online documentation].<br>
A pandas series is a 1 dimensional array with named keys.<br>
Pandas Series have all kind of methods similar to [[Numpy]] like main, std, min, max,.... In fact Pandas is using numpy to do this.
;s = pd.Series([])
;s = pd.Series([valuelist],[indexlist])
:Initialize a series. If indexlist is omitted the keys are integers starting at 0.
;s[<key>] = <value>
:Assign <value> to the series element with key <key>
:The order in the series is the order in which they are created, NOT the numeric order.
:Elements can be addressed as <code>s[<key>]</code>, <code>s.<key></code> or <code>s[<numkey>]</code>. Where <numkey> is defined by the order the element was created.
:Once you have used named keys in a series you cannot create new elements with a numeric key.
;s.index
:All indexes in the series. Can be sliced to find a particular index.
;s.describe()
:Series statistics

All in 1 example:
<syntaxhighlight lang=python>
import numpy as np
import pandas as pd
s = pd.Series([])
for i in range(50):
s[i] = int(np.random.random() * 100)

for i in s.index:
print(i,s[i])
</syntaxhighlight>

Funny, you can do <code>s[0]</code> but not
<syntaxhighlight lang=python>
for i in s:
print(s[i])
</syntaxhighlight>
To get all values from the series you do:
<syntaxhighlight lang=python>
for v in s:
print(v)
</syntaxhighlight>
To get the indexes too:
<syntaxhighlight lang=python>
for i in s.index:
print(i,s[i])
</syntaxhighlight>

==TimeSeries==
Indexes that contains datetime values are automatically casted to a DatetimeIndex.

;df.resample('5D').mean
:Return the dataframe with the average value of each 5 days.

;<nowiki>df.resample('1H').agg({'ColumnA':'first','ColumnB':'max', 'ColumnC':'min', 'ColumnD':'last' })</nowiki>
:Resample with aggregation specified per column.

;df.resample('5D',origin='end')
:Resample so the last timestamp is used, by default it starts with the first, the last set may be incomplete.

==Datetime functions==
;df[datefield] = pd.to_datetime(df[datefield],format='%d-%m-%Y')
:Convert a string-column with a date into datetime. Format is the input format.
:WARNING: without format to_datetime tries to make up what the format is. This is pretty intelligent but can easily be wrong with day-numbers under 13
;df[datefield] = pd.to_datetime(df[datefield],unit='s')
:Convert a column with [[unix time]] into datetime.
;df[<timefield>] = testdf[<datetime-field>].dt.time
:Return a series (column) with the time-value of a datetime-column

=Reading Data=
;read_html(url)
:Read html tables into a list of [[#DataFrame |dataframes]] (no header, no index)
Example code. The first line in the table is a header, the first column the index (e.g. dates), decimal specifies the decimal point character.
<syntaxhighlight lang=python>
tables = pd.read_html(url,header=0,index_col=0,decimal=<char>)
</syntaxhighlight>

;read_sql(query,cnx,index_columns=[col1,col2])
:Read data from the database opened on cnx (see [[Python:Databases]])
<syntaxhighlight lang=python>
df = pd.read_sql(query,cnx,index_col=['Primarykey'])
</syntaxhighlight>

;read_csv(csvfile)
:Read data from a file with Character Separated Values
;<nowiki>read_csv(csvfile, delimiter=',', usecols=[<columnA>, <columnB>], dtype = {<columnA>: str})</nowiki>
:Read columns <columnA> and <columnB> into a dataframe and force columnA to string type.
;<nowiki>read_csv(csvfile, index_col=False, header=None, names=['columnA', columnB'], na_values = 'None')</nowiki>
:Read from a file with no indexcolumn nor a headerrow, set the colomnnames and handle 'None' and NA value.

==Excel==
;read_excel(xlsfile, sheetname)
:Read data from the sheet with 'sheetname' in a microsoft [[XLS|Excel file]] into a dataframe
;<nowiki>read_excel(xlsfile,sheetname,converters={'columna':str,'columnb':str})</nowiki>
:Force columns to be read as string
;read_excel(xlsfile, sheet_name=None)
:Read all sheets into a dict {sheetname: df,...}. Sheets can be specified by number or name, if multiple, in a list. If only 1 sheet is specified a dataframe is returned, not a dict of dataframes.
<syntaxhighlight lang=python>
dfdict = pd.read_excel(filename, sheet_name=None)
for sheetname, df in dfdict.items():
print(f"Working on sheet {sheetname}")
for index, row in df.iterrows():
print(row)
adict = df.to_dict('records') # Creates a list of dicts [{columname: value,...},...]
print(json.dumps(adict, indent=4))
</syntaxhighlight>

Convert all sheets to CSV files
<syntaxhighlight lang=python>
import sys
import pandas as pd

def main():
for filename in sys.argv[1:]:
try:
dfdict = pd.read_excel(filename, sheet_name=None)
except Exception as exception:
print(f"{filename} cannot be read by pandas.read_excel as {exception}")
else:
for sheetname, df in dfdict.items():
print(f"Working on {filename}, {sheetname}")
if len(dfdict) == 1:
sheetname = ''
else:
sheetname = f"_{sheetname}"
lines = df.to_dict('records')
header = lines[0].keys()
outputfile = filename.replace('.xlsx', f"{sheetname}.csv")
fh = open(outputfile, 'w')
fh.write(','.join(header) + '\n')
for line in lines:
fh.write(','.join(map(str,line.values())) + '\n')
fh.close()
return

main()
</syntaxhighlight>

Pandas

2026-08-04T07:01:57Z

Hdridder: /* Use data */

[[Category:Python]]
Check the [https://pandas.pydata.org/pandas-docs/stable/10min.html 10 minutes to Pandas] too or [https://pythonexamples.org/pandas-examples/ pythonexamples].
;import pandas as pd
:Import the library, we assume this was done on this page

=DataFrame=
Object for tabular data (that is e.g. obtained by read_html).

Each column is a [[Pandas#Series]]

;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}])</nowiki>
:Create a simple dataframe having a range index, 2 rows and 2 columns
;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}],[row1,row2])</nowiki>
:Create a dataframe having a list index. The size of index must match the number of rows.

Single columns can be addressed by there name like in <code>df.columnname.unique()</code> to get unique values in a column.

Single or multiple columns can be addressed as list like in <code>df.set_index([column1,column2])</code> to set an index over 2 columns.

;df.copy
:Make a copy of df ( = will point to the same dataframe, like with [[Python:DataTypes#Dictionary or dict|dictionaries]])

;df.join(table2)
:Like SQL-join merge on index
;df.merge(table2,on=<column>)
:Like SQL-join on <column> = df.index
;dfall.merge(table2, on = <column>, how='left', indicator=True)
:Return df joined with table2. Add column _merge with value 'left_only' to records that only exist in df.
;df = dfall[dfall['_merge'] == 'left_only']
:Return dataframe with all records not in table2
;df = pd.concat(dflist)
:Concatenate all dataframes in dflist

==Information==

;df.info()
:Information on dataframe (Index, size, datatypes per column)
;df.size
:Return the number of cells in a dataframe
;df.shape
:Return the number of rows and columns in a dataframe as [[Python:DataTypes#Tuple|tuple]].
;df.describe()
:Return various attributes for numeric columns in the dataframe. This may indicate if there are outlyers.
;df.head(x)
;df.tail(x)
:Return first/last x data rows of df (5 is the default value for x).

;df.index
:Return the table index (class = pandas.core.indexes.base.Index)

;df.columns
:Return the column headers (class = pandas.core.indexes.base.Index)
;df.dtypes
:Return the data-type of the columns
;df.columnname.dtype
:Return the data-type of column columnname

==Modify==

;<nowiki>df.rename(columns={oldname:newname}, inplace=True)</nowiki>
:Redefine the column headers (modifies the dataframe itself, nothing is returned)

;df.set_index(['col1','col2'])
:Return the dataframe with a new index composed of 2 columns
;df.reset_index()
:Return the dataframe with a default index (range)
;df.sort_index(ascending=False)
:Sort the dataframe on the index. 'True' is the default if 'acsending' is not supplied.
;df.index = df.index.map(str)
:For a data-type on the index

;df.drop(index)
;df.drop([indexes])
;df.drop(range(0,3))
:Remove row(s) from a dataframe.
;df.drop(columns=[listofcolumnstodrop]
:Remove columns from a dataframe.

;df.dropna(thresh=x)
:Return only rows that have at least x columns with a none-NA value. If x < 2 the threshold can be dropped.

;df.loc[<index>,['field1','field2']] = [value1,value2]
:Change values for index <index> in a dataframe
;df.fillna(<value>)
;df[[<column1>,<column2>]] = df[[<column1>,<column2>].fillna(<value>)
:Return dataframe with all NA-values (in the selected columns) replaced by <value>

;table.agg(newname=('columname', np.max))
:Add newname to df with the max of columnname. This named aggregations is only supported from version v0.25

;df.transform(<function>)
:Apply a function to an existing column or the entire dataframe.

;df.assign(newcolumn = <expression>)
:Add column newcolumn to the dataframe
:Expression can be a function like "lambda d: d['column'] * <something>"

;df.apply(<function>)
:Apply a function to the dataframe. By default every element of the table.
;<nowiki>df.apply(lambda x: <function>(x) if x.name in [columnname] else x)</nowiki>
:Apply a function to specified colums
;<nowiki>df[columname] = df[columname].apply(lambda x: <function>(x))</nowiki>
:Change a column in the dataframe by applying <function>

;df.replace(<pattern>,<newvalue>,regex=True)
:Replace <pattern> with <newvalue> in the entire dataframe

;df[columna].mask(df[columna] == origvalue , newvalue, inplace=True)
:Change all cells in columa that have origvalue into newvalue (inplace, so in the dataframe itself).

==Select data==
Use .loc, else colomn-names will be considered too (if I understand this[https://stackoverflow.com/questions/38886080/python-pandas-series-why-use-loc] correctly).

;WARNING!!
:at least with .loc if only 1 item is found is it returned as string(/float?) else as dataframe/series.

;df.[[iloc]][<slice>]
:Return the rows indicated by <slice> as pandas.Series
;<nowiki>df.iloc[:,0:3]</nowiki>
;<nowiki>df[[col0,col1,col2]]</nowiki>
:Return 3 columns from the dataframe
;df.filter(regex=<regex>,axis='columns')
:Return all columns which name matches <regexp>. (axis=1)

;df.loc[<index>]
:Select the row(s) with index <index>
;df.loc[<indexname>].<columnname>
;df.loc[<indexname>].[<columnname>]
:Select only 1 column from a row as pandas series. You may convert that to a list or set immediately by using <code>list(df.loc[<indexname>].<columnname>)</code>
;df.loc[0][0]
:Select a cell in a dataframe with no index or header
;<nowiki>df.loc[lambda d: d[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1].notnull()]</nowiki>
:Select all rows where column1 equals <value> or do not have a null-value.
;<nowiki>df.loc[(df[column1] == <value>) | (df[colomn1] == <value2>)]</nowiki>
:Combine conditions with '|' (or) or '&' (and). Use () on each condition!!
;<nowiki>df.loc[datetime.today()-pd.Timedelta(days=7) : datetime.today()]</nowiki>
:Select all rows where the (datetime)-index is between 7 days ago and today.
:NOTE!! The index must be sorted in ascending order[https://stackoverflow.com/q/67298584/2213323].
;df.last('7D')
:Select the last 7 days in the dataframe. For this als the datetime index must be sorted in ascending order

See also the [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.contains.html#pandas.Series.str.contains pandas series pages]
;df[df['fieldname'].str.contains('regexp',regex=True)]
:Select all rows in df where fieldname contens matches regexp. The field must have a string datetype. If you are looking for a simple string you can leave out the regex=True part.
;df[df['fieldname'].gt(value)]
:Select all rows in df where fieldname is bigger than value. The field must have a numeric datetype

;df.filter(regex=<regex>,axis='index')
;df.filter(regex=<regex>,axis='index').<columnname>
;df.filter(regex=<regex>,axis='index').index
:Return all rows for which in index matches <regexp> or get only the column of the matched indexes. (axis=0 ) or the indexname(s).
:The .index returns the matching indexes as a [[Python:DataTypes#List|list]].

;df.sort_values(<columnname>)
;df.sort_values([<column1>,<colunm2>],ascending=(True,False))
:Return the dataframe sorted on the values in the columns. The second form sorts on column1 first and then on column2, column1 ascending, column2 descending
;df.groupby([column1,column2])
:Return a DataFrameGroupBy object, this is not a dataframe
;df.collname.unique()
:Return a [[Numpy]] array with all distinct values in column 'collname'

;Get all rows from df1 that have an index that exists in df2
df = df1[df1.index.isin(df2.index)]

;Or get all intersectioned indices and select by loc:
df = df1.loc[df1.index & df2.index]
df = df1.loc[np.intersect1d(df1.index, df2.index)]
df = df1.loc[df1.index.intersection(df2.index)]

==Use data==
;Print 2 attributes for each row and check for no value
:pd.isna(value) works for Null, NaN (Nod a number) and NaT (not a time) too.
<syntaxhighlight lang=python>
for index, row in df.iterrows():
print(row['ColumnA'],row['ColumnB'])
if ps.isna(row['ColumnA']):
print('Found a NaN value')
</syntaxhighlight>
;In multi-column indexes the index is returned as a tuple
<syntaxhighlight lang=python>for (indexA, indexB), row in df.iterrows():</syntaxhighlight>

;Print all values for the first row
<syntaxhighlight lang=python>
for column, row in df.iteritems():
print(column,row[0])
</syntaxhighlight>

;Loop over columns
<syntaxhighlight lang=python>
for column in df:
print(f"Columnname {column}")
</syntaxhighlight>

;df.iloc[0]
:return the values of the first row as pandas series. If there is only 1 column returns the value.

;df.iloc[0][0]
:return the value of the first column in the first row. Columns can also be identified by name

;df[<column>].quantile(q=0.1)
:Return the 10th percentile (10% of the values in <column> is lower than the 10th percentile)

;df[<column].str.extract(r"A regex(p\w+)")
:Return a series (column) of the values selected by [[Regular Expressions]] (p\w+)
;df[<columna>].map(str) + '_' + df[<columnb>].map(str)
:Return a series (column) with the columns concatenated. If not mapped to str an error will be thrown.
;<nowiki>df[<column>].map(str).apply(lambda x: x[-1])</nowiki>
:Return a series (column) with the last character of the <column>

=Series=
Pandas Series [https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html online documentation].<br>
A pandas series is a 1 dimensional array with named keys.<br>
Pandas Series have all kind of methods similar to [[Numpy]] like main, std, min, max,.... In fact Pandas is using numpy to do this.
;s = pd.Series([])
;s = pd.Series([valuelist],[indexlist])
:Initialize a series. If indexlist is omitted the keys are integers starting at 0.
;s[<key>] = <value>
:Assign <value> to the series element with key <key>
:The order in the series is the order in which they are created, NOT the numeric order.
:Elements can be addressed as <code>s[<key>]</code>, <code>s.<key></code> or <code>s[<numkey>]</code>. Where <numkey> is defined by the order the element was created.
:Once you have used named keys in a series you cannot create new elements with a numeric key.
;s.index
:All indexes in the series. Can be sliced to find a particular index.
;s.describe()
:Series statistics

All in 1 example:
<syntaxhighlight lang=python>
import numpy as np
import pandas as pd
s = pd.Series([])
for i in range(50):
s[i] = int(np.random.random() * 100)

for i in s.index:
print(i,s[i])
</syntaxhighlight>

Funny, you can do <code>s[0]</code> but not
<syntaxhighlight lang=python>
for i in s:
print(s[i])
</syntaxhighlight>
To get all values from the series you do:
<syntaxhighlight lang=python>
for v in s:
print(v)
</syntaxhighlight>
To get the indexes too:
<syntaxhighlight lang=python>
for i in s.index:
print(i,s[i])
</syntaxhighlight>

==TimeSeries==
Indexes that contains datetime values are automatically casted to a DatetimeIndex.

;df.resample('5D').mean
:Return the dataframe with the average value of each 5 days.

;<nowiki>df.resample('1H').agg({'ColumnA':'first','ColumnB':'max', 'ColumnC':'min', 'ColumnD':'last' })</nowiki>
:Resample with aggregation specified per column.

;df.resample('5D',origin='end')
:Resample so the last timestamp is used, by default it starts with the first, the last set may be incomplete.

==Datetime functions==
;df[datefield] = pd.to_datetime(df[datefield],format='%d-%m-%Y')
:Convert a string-column with a date into datetime. Format is the input format.
:WARNING: without format to_datetime tries to make up what the format is. This is pretty intelligent but can easily be wrong with day-numbers under 13
;df[datefield] = pd.to_datetime(df[datefield],unit='s')
:Convert a column with [[unix time]] into datetime.
;df[<timefield>] = testdf[<datetime-field>].dt.time
:Return a series (column) with the time-value of a datetime-column

=Reading Data=
;read_html(url)
:Read html tables into a list of [[#DataFrame |dataframes]] (no header, no index)
Example code. The first line in the table is a header, the first column the index (e.g. dates), decimal specifies the decimal point character.
<syntaxhighlight lang=python>
tables = pd.read_html(url,header=0,index_col=0,decimal=<char>)
</syntaxhighlight>

;read_sql(query,cnx,index_columns=[col1,col2])
:Read data from the database opened on cnx (see [[Python:Databases]])
<syntaxhighlight lang=python>
df = pd.read_sql(query,cnx,index_col=['Primarykey'])
</syntaxhighlight>

;read_csv(csvfile)
:Read data from a file with Character Separated Values
;<nowiki>read_csv(csvfile, delimiter=',', usecols=[<columnA>, <columnB>], dtype = {<columnA>: str})</nowiki>
:Read columns <columnA> and <columnB> into a dataframe and force columnA to string type.
;<nowiki>read_csv(csvfile, index_col=False, header=None, names=['columnA', columnB'], na_values = 'None')</nowiki>
:Read from a file with no indexcolumn nor a headerrow, set the colomnnames and handle 'None' and NA value.

==Excel==
;read_excel(xlsfile, sheetname)
:Read data from the sheet with 'sheetname' in a microsoft [[XLS|Excel file]] into a dataframe
;<nowiki>read_excel(xlsfile,sheetname,converters={'columna':str,'columnb':str})</nowiki>
:Force columns to be read as string
;read_excel(xlsfile, sheet_name=None)
:Read all sheets into a dict {sheetname: df,...}. Sheets can be specified by number or name, if multiple, in a list. If only 1 sheet is specified a dataframe is returned, not a dict of dataframes.
<syntaxhighlight lang=python>
dfdict = pd.read_excel(filename, sheet_name=None)
for sheetname, df in dfdict.items():
print(f"Working on sheet {sheetname}")
for index, row in df.iterrows():
print(row)
adict = df.to_dict('records') # Creates a list of dicts [{columname: value,...},...]
print(json.dumps(adict, indent=4))
</syntaxhighlight>

Convert all sheets to CSV files
<syntaxhighlight lang=python>
import sys
import pandas as pd

def main():
for filename in sys.argv[1:]:
try:
dfdict = pd.read_excel(filename, sheet_name=None)
except Exception as exception:
print(f"{filename} cannot be read by pandas.read_excel as {exception}")
else:
for sheetname, df in dfdict.items():
print(f"Working on {filename}, {sheetname}")
if len(dfdict) == 1:
sheetname = ''
else:
sheetname = f"_{sheetname}"
lines = df.to_dict('records')
header = lines[0].keys()
outputfile = filename.replace('.xlsx', f"{sheetname}.csv")
fh = open(outputfile, 'w')
fh.write(','.join(header) + '\n')
for line in lines:
fh.write(','.join(map(str,line.values())) + '\n')
fh.close()
return

main()
</syntaxhighlight>

Python:Control structures

2026-07-29T09:55:03Z

Hdridder:

[[Category:Python]]

Most flow control is standard. Indentation defines the code block to execute. You must be strict, all indents must be the same in a block, a mix of spaces and tabs is not allowed (unless each line uses the same mix).

You can define you own [[Python:Functions]] too.

<syntaxhighlight lang=python>
if expression:
block
elif expression:
block
else:
otherblock

for var1 in itterable: # keys if itterable is a dict
block
if somethinghappens:
break

while expression:
block
if somethinghappens:
continue
</syntaxhighlight>

;<code>break</code>
:Stop looping immediately. Jump to the next statement after the loop.
;<code>continue</code>
:Stop the current iteration immediately and jump the the top of the loop, do the next iteration.
;<code>pass</code>
:No operation, use e.g. to avoid negative tests
;<code>var1 = ( x if <condition1> elif <condition2> y else z )</code>
:Different way to assign a value conditionally. () only needed if spread over more lines.

==Exception handling==

Exceptions are used a lot by python programmers. The mantra is: "Easier to ask for forgiveness than permission" aka EAFP; and it is said to be faster.

To catch exceptions:
<syntaxhighlight lang=python>
try:
blockthatmaythowexception
except <exception>:
blockifexceptionisthrown
except <exception2> as ex:
blockifexception2isthrown
else:
blockif_NO_exceptionisthrown
</syntaxhighlight>
NOTE: elif is not allowed here.

Specifying a specific exception is optional but highly recommended. However, take care you don't forgive real faults.

If more exceptions are defined with 1 try only 1 exception block will be executed,

For reporting where the exception happened use [[#traceback]].

;except <exception> as ex
:Puts the exception result in ex. ex.message has the reason for the exception.

;type(ex)
:The exception class

;except Exception
:Catches any exception

;except NameError
:The variable does not exist

;except KeyError
:The key does not exist in the dict

;except IndexError
:The index does not exist in the list

A full list of exception-classes can be found in the [https://docs.python.org/3/library/exceptions.html#exception-hierarchy python docs]

;raise <Exception>(<text>)
:Raise an <Exception> exception, <Exception>-class must exist, <text> is up to you.

===Traceback===
Show where things went wrong
<syntaxhighlight lang=python>
import traceback
tracetext = traceback.format_exc() # Return info as string
traceback.print_exc() # Print the info
</syntaxhighlight>

==Warnings==
[https://docs.python.org/3/library/warnings.html Warnings] are a subcategory of Exceptions for problems that do not break the script. E.g. for deprecated modules, division by 0 etc.

After importing the warnings package you can do things like
;warnings.simplefilter(action='ignore', category=UserWarning)
:Ignore warnings in a specific category.
;warnings.filterwarnings("error")
:Turn warnings into exceptions
;warnings.resetwarnings()
:Reset to the default handling of warnings

Example:
<syntaxhighlight lang=python>
import warnings
...
...
warnings.simplefilter(action='ignore', category=UserWarning)
<code that throws warning>
warnings.resetwarnings()
</syntaxhighlight>

This way the warning settings are restored after the known one has been passed

Iptables

2026-07-06T09:50:41Z

Hdridder:

[[category:networking]]
[[category:Linux/Unix]]
[[nftables]] is a successor of iptables of firewall managment

;iptables -F
:Flush all rules
;iptables -nvL
:Show all rules

In Debian rules are in:

<code>/etc/iptables/rules.v4</code><br>
<code>/etc/iptables/rules.v6</code>

Pretty strict rules for an IPv4 webserver can be configured like this:
<syntaxhighlight lang=bash>
cat /etc/iptables/rules.v4
*filter
:INPUT DROP -4 [0:0]
:FORWARD DROP -4 [0:0]
:OUTPUT ACCEPT -4 [0:0]
-A INPUT -4 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -4 -p icmp -j ACCEPT
-A INPUT -4 -i lo -j ACCEPT
-A INPUT -4 -p tcp -m conntrack --ctstate NEW -m tcp -s 192.168.2.0/24 --dport 22 -j ACCEPT
-A INPUT -4 -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -4 -p tcp -m conntrack --ctstate NEW -m tcp --dport 443 -j ACCEPT
COMMIT

cat /etc/iptables/rules.v6
*filter
:INPUT DROP -6 [0:0]
:FORWARD DROP -6 [0:0]
:OUTPUT DROP -6 [0:0]
-A INPUT -6 -i lo -j ACCEPT
-A OUTPUT -6 -o lo -j ACCEPT
COMMIT
</syntaxhighlight>

Activated by <code>iptables-restore < /etc/iptables/rules.v4</code> this results in:

<syntaxhighlight lang=bash>
iptables -nvL
Chain INPUT (policy DROP 100 packets, 27924 bytes)
pkts bytes target prot opt in out source destination
19748 1436K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
25 1800 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
84 5040 ACCEPT tcp -- * * 192.168.2.0/24 0.0.0.0/0 ctstate NEW tcp dpt:22
2 92 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:80
9 540 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW tcp dpt:443

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 35665 packets, 49M bytes)
pkts bytes target prot opt in out source destination
</syntaxhighlight>
For IPv6
<syntaxhighlight lang=bash>
ip6tables-restore < /etc/iptables/rules.v6

ip6tables -nvL
Chain INPUT (policy DROP 14 packets, 2902 bytes)
pkts bytes target prot opt in out source destination
13 2476 ACCEPT all lo * ::/0 ::/0

Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy DROP 16 packets, 2134 bytes)
pkts bytes target prot opt in out source destination
13 2476 ACCEPT all * lo ::/0 ::/0

</syntaxhighlight>

=ufw=
(Uncomplicated FireWall) is a frontend to iptables that creates a pretty complicated set of firewall rules.

Configuration files are stored in /etc/ufw/

By default the INPUT and FORWARD chain drop all, OUTPUT allow all.

;ufw enable/disable
:Enable/Disable the firewall rules

;ufw allow proto tcp from 192.168.1.0/24 to any port 22;
:ufw allow proto tcp from fd8e:xxx:xxx:xxx::/64 to any port 22
:Add firewall rule to allow port 22 (ssh) from a local subnet to the current set and to the ufw configuration files. Works for IPv6 addresses too

;ufw delete allow proto tcp from 192.168.1.0/24 to any port 22
:Remove firewall rule to allow port 22 (ssh) from a local subnet from current set and from the ufw configuration files.

;ufw delete <rulenumber>
:Delete the rule with number <rulenumber> (see below)

;ufw status numbered
:Show rule summary with rule numbers

MariaDB/MySQL

2026-06-09T19:50:47Z

Hdridder:

[https://mariadb.org/ MariaDB] is an open source fork of [http://www.mysql.com/ MySQL]. There are some differences but most are internal, for users the commands and [[SQL]] syntax is exactly the same.

[https://www.phpmyadmin.net/ phpMyAdmin] is a web based GUI for the maintenance of MariaDB and MySQL servers.

=Performance related queries=
;show status like 'Qcache_total_blocks'
:Fragmentation of the query cache (lower is better)
;flush query cache
:Flush the query cache to fix the fragmentation (check if cache sizes need to be changed too)

=User management=
User data is stored in table mysql.user
;alter user username@host identified by 'password'
:Change a users password
;drop user username@host
:Delete a user
;grans all privileges on *.* to 'user'@'host';
:Grant all privileges to all database to a user

=Login to password user without plain text password on commandline=
Put the password in a protected config file.

vi ~/.my.conf
[client]
password=<password>

chmod 600 ~/.my.cnf

When you want to use a different filename start mysql-clinet like
mysql --defaults-file=/home/francis/mysql-opts

NOTE: commands can be stored in the history logging. Make sure the log-file (~/.mysql_history) has very restrictive permissions

=Move users from one server to another=

Starting on the source server execute this script
<syntaxhighlight lang=bash>
PASSWORD=$1
mysql -B -N -uroot -p${PASSWORD} -e "SELECT CONCAT('\'', user,'\'@\'', host, '\'') FROM user WHERE user IN ('username1', 'username2', 'username3')" > users.txt
while read line
do
mysql -B -N -uroot -p${PASSWORD} -e "SHOW GRANTS FOR $line"
done < users.txt > users.sql
sed -i 's/$/;/' users.sql
</syntaxhighlight>

Move the file to the other server and there do:

<code>mysql -u root -p < users.sql</code>

=Set up replication=
With replication all transactions on the master database server are immediately replicated to the slave database server. So you create a hot copy of the database that can be used in case of a failure of the master server or to offload the master server by executing retrieval queries on the slave.

The slave server logs into the master and checks if the bin-log position differs, if so all transactions after the last binlog-position are redone on the slave.

This procedure is for a large part based on a tutorial found on howtoforge.com [https://www.howtoforge.com/mysql_database_replication]

For multi master replication to 1 slave check [https://mariadb.com/kb/en/multi-source-replication/ this].

==On the master==
;Edit the my.cnf so it has below lines.
:The server must be accessible from the slave server and the server-id must be unique in you network
:The binlog-do-db specifies for which database the binlog is written, one line per database. If omitted for all databases binlog is written except those specified by binlog-ingnore-db.
#skip-networking
#bind-address = 127.0.0.1
log-bin = /var/log/mysql/mysql-bin.log
#binlog-do-db=exampledb
binlog-ignore-db = mysql
binlog-ignore-db = information_schema
binlog-ignore-db = performance_schema
binlog-ignore-db = phpmyadmin
server-id=1

;Restart the database server
/etc/init.d/mysql restart

;Create a replication user
mysql -uroot -p
mysql> GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY '<some_password>';
mysql> FLUSH PRIVILEGES;

;Make sure nothing is changes in the database during backup
mysql> USE exampledb;
mysql> FLUSH TABLES WITH READ LOCK;
Or, for all databases;
mysql> FLUSH TABLES WITH READ LOCK;
mysql> SET GLOBAL read_only = 1;
;Check status;
mysql> SHOW MASTER STATUS;
+---------------+----------+--------------+------------------+
| File | Position | Binlog_do_db | Binlog_ignore_db |
+---------------+----------+--------------+------------------+
| mysql-bin.006 | 183 | | |
+---------------+----------+--------------+------------------+
1 row in set (0.00 sec)

;Dump the database:
mysqldump -u root -p<password> --opt exampledb > exampledb.sql

;Unlock the tables
mysql -u root -p<password>
mysql> UNLOCK TABLES;
mysql> quit;
Or, for all databases;
mysql> SET GLOBAL read_only = 0;
mysql> UNLOCK TABLES;

==On the slave server==

;Create the slave database
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CREATE DATABASE exampledb;
quit;
</syntaxhighlight>

;Load the database dump created on the master
<code>mysql -u root -p<password> exampledb < /path/to/exampledb.sql</code>

;Edit the my.cnf so it has these lines
:The server-id must be unique in you network
:The replicate-do-db specifies for which database the binlog on the master is read, one line per database. If omitted for all databases binlog is read.
<syntaxhighlight lang=text>
server-id=2
replicate-do-db=exampledb
</syntaxhighlight>

;Restart the database server
<code>/etc/init.d/mysql restart</code>

In the following statement the values from SHOW MASTER STATUS above must be used.

<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
STOP SLAVE;
CHANGE MASTER TO MASTER_HOST=<masterIP>, MASTER_USER='slave_user', MASTER_PASSWORD='<password>', MASTER_LOG_FILE='mysql-bin.006', MASTER_LOG_POS=183;
START SLAVE;
quit;
</syntaxhighlight>

====In case of using GTID for replication====
<code>mysql -u root -p<password></code>
<syntaxhighlight lang=sql>
CHANGE MASTER TO master_host= <masterIP>', master_user='<slave_user>', MASTER_PASSWORD='<password>', master_use_gtid=current_pos;
</syntaxhighlight>

;<code>SELECT @@GLOBAL.gtid_slave_pos;</code>
:Show current GTID position
;<code>SET GLOBAL gtid_slave_pos = ""</code>
:Reset counter, slave databases must be in sync with the master.

=Add a database to replication=

Everything must be working fine and log positions up to date. On the slave Read_Master_Log_Pos is the same as the Exec_Master_Log_Pos and the Read_Master_Log_Pos is the same as the masters Position (in show master status).

Then you can do:
* Slave: Add database to my.cnf (replicate-do-db=<dbname>)
* Slave: <code>stop slave;</code>
* Master: <code>flush tables with read lock;</code>
* Master: dump database
* Master: <code>unlock tables</code>
* Slave: Load the database dump
* Slave: Restart the database server
Then check the slave status <code>show slave status \G</code>

=Repair replication=
==On the slave server==
;Stop replication processes
mysql -u root -p
mysql> stop slave;

==On the master server==
Dump the replicated databases including the replication data.
<code>mysqldump --databases <db1> <db2> --master-data=1 > ../mysql_dump_$(hostname)_$(date +%Y-%m-%d_%H-%M).sql.dump</code>

==On the slave server==
;Import the databases
<code>mysql -uroot -p < <dumpfile></code>
;Restart the slave processes and check the slave status.
When needed update the slave log-file and position as in the masters <code>show slave status</code>
mysql> CHANGE MASTER TO MASTER_LOG_FILE='<filename>', MASTER_LOG_POS=<postition>;
Then:
mysql> start slave;
mysql> show slave status \G

This should not show any errors

=Populate and update timezone information=
Timezone information is required to use functions like CONVERT_TZ. The mysql.time_zone table need to be populated and maintained when the data changes.

On a [[synology]] server following command takes care of this:

<code>/volume1/@appstore/MariaDB10/usr/local/mariadb10/bin/mysql_tzinfo_to_sql /usr/share/zoneinfo |mysql -uroot -p mysql</code>

Webdesign

2026-06-07T08:53:54Z

Hdridder: /* Certificate */

[[category:Webdesign]]
=Speed=
;Tests
*[https://developers.google.com/speed/pagespeed/insights/ Google]
*[https://tools.pingdom.com/ Pingdom]

;Enable [https://en.wikipedia.org/wiki/HTTP/2 http/2] on port 443 for better performance
:In your webserver condfig put <code>listen 443 http2 ssl;</code>

=HTML=
Make up language for web pages.
Check [https://www.w3schools.com/ w3schools]

;<a href="a url" target="_blank">go to a url</a>
:Hyperlink to a url to be opened in a new tab. Display "go to a url" as hyperlink (underscored)

=CSS=
Change presentation style for html elements.
Check [https://www.w3schools.com/ w3schools]

;Use a lightyellow background, black characters and readable font:
<syntaxhighlight lang=css>
body {
color: #000000;
background-color: #FFFFDD;
font-family: arial, helvetica, sans-serif;
}
</syntaxhighlight>

=Colors=
Check [https://html-color.codes/ 🎨 HTML Color Codes]

=Certificate=
Check [https://www.ssl.org/ ssl org]

Webdesign

2026-06-07T08:49:20Z

Hdridder:

DNS

2026-05-18T15:13:51Z

Hdridder:

[[Category:Networking]]
[[Category:Linux/Unix]]

Domain Name Service. Bind a name to IP-addresses.

=Server side=
Zone transfer between master and slave server is on TCP port 53.

Disable resolving IPv6 addresses by editing /etc/default/bind9 to have <code>OPTIONS="-4 -u bind"</code>

=Client side=
Clients query the nameserver on UDP port 53.

;host <hostname or IPaddress>
:Resolve hostname to IPadddres or IPaddress to hostname
;host <host or IP> <nameserver>
:Resolve using the specified nameserver

;dig <hostname> @<nameserver>
:Resolve hostname with extra info from the optinoal <nameserver>

;/etc/resolv.conf
<syntaxhighlight lang=text>
options timeout:1 attempts:1 rotate
nameserver <dns-server1 IP>
nameserver <dns-server2 IP>
options <interface>
search <default search domain>
</syntaxhighlight>

;If the systemd-resolved is active
/etc/resolv.conf has 127.0.0.53 as nameserver

;resolvectl status
:Show which nameservers are used by the systemd-resolved

DNS

2026-05-18T15:11:59Z

Hdridder: /* Client side */

Domain Name Service. Bind a name to IP-addresses.

=Server side=
Zone transfer between master and slave server is on TCP port 53.

Disable resolving IPv6 addresses by editing /etc/default/bind9 to have <code>OPTIONS="-4 -u bind"</code>

=Client side=
Clients query the nameserver on UDP port 53.

;host <hostname or IPaddress>
:Resolve hostname to IPadddres or IPaddress to hostname
;host <host or IP> <nameserver>
:Resolve using the specified nameserver

;dig <hostname> @<nameserver>
:Resolve hostname with extra info from the optinoal <nameserver>

;/etc/resolv.conf
<syntaxhighlight lang=text>
options timeout:1 attempts:1 rotate
nameserver <dns-server1 IP>
nameserver <dns-server2 IP>
options <interface>
search <default search domain>
</syntaxhighlight>

;If the systemd-resolved is active
/etc/resolv.conf has 127.0.0.53 as nameserver

;resolvectl status
:Show which nameservers are used by the systemd-resolved

Python:DataTypes

2026-05-13T11:48:45Z

Hdridder: /* list */

[[Category:Python]]

=Object Classes=
Lots of things to tell about strings, they have there own [[Python:Strings]] page.

For different number types we have [[Python:Numbers]].

Also for numpy (module for scientific arithmetic) there is a special [[Numpy]] page.

Objects are iterable if they can contain more than 1 ordered objects ([[Python:Strings|string]], list, tuple, dict).

Objects are mutable if their content can be changed (list, set, dict)

;isinstance(<obj>, <class>)
:Boolean (returns True or False) to check if <obj> is an instance of <class>. Classnames are str, int, dict, list, set, tuple, range, ...

locals() and globals() are dictionaries holding the global and local variables so you can do:
;<code>if var in locals()</code>
:Check if a variable exists as local
;<code>if var in globals()</code>
:Check if a variable exists and is global.
;globals().update(adict)
:updates the globals with the dict. You can now address the keys as variable.

<strong>NOTE:</strong> Variables are pointers to objects, not the object itself. dict2 = dict1 does not make a new dictionary. Both variables point to the same dictionary. To make a copy use dict2 = dict(dict1). Same for lists, sets and tuples.

==list==
Class of iterable, mutable objects. Lists can be compared to arrays in other languages. Lists can contain a mixture of all kind of objects.

;lst1 = []
:Initialize an empty list

;lst1 = [0] * 10
:Create a list with 10 elements 0

;lst1.append(2)
:Add the '2' object to the end of lst1

;lst1.extend(list2)
;lst1 + lst2
:Add the elements of lst2 to the end of lst1

;lst1.pop(n)
:Remove and return n<sup>th</sup> element from lst1. Last element if n is not specified.
;lst1.remove(x)
:Remove object x from the list.

;lst1 = list(object)
:Convert object to a list (object is e.g. set, tuple or string)

;count = lst1.count[x]
:Return the number of occurrences of x in lst1

;lst1.index(x)
:Return the first position in lst1 where x is found
;<nowiki>len(lst1) - 1 - lst1[::-1].index(x)</nowiki>
:Return the last position in lst1 where x is found
;bisect(lst1, x)
:Return index of first value higher than x. bisect needs to be imported as <code>from bisect import bisect</code>

;lst1.sort()
:Sort lst1 and return 'None' object

;lst2 = sorted(iterable, key=int)
:Return the iterable object sorted as list.
:The optional key= part converts the elements to int before sorting. Other functions can be used too.

;<nowiki>srtlist = sorted(alist, key = lambda x: (x[4], x[0]))</nowiki>
:Return alist sorted on the third and the first element of each list in alist (alist is a list of lists)

[https://wiki.python.org/moin/HowTo/Sorting More on sorting].

;random.shuffle(alist)
:Randomize the list element order. The list itself is changed, nothing is returned.

;print (','.join(map(str, alist)))
:Concatenate the elements of alist to <element1>,<element2>,... and print
;print ('\n'.join(map(str, alist)))
:Print alist with each element on another line
;list(map(int, alist))
:Return a list with all elements in alist mapped to integer.

;Put items matching a regular expression in newlist.
More on regular expressions in [[Python:Strings#Regular_Expressions_(regexp)]]

NOTE: filter applies a function (regexp matching in this case) on the object (list1). The object is modified. A filter object is returned. A filter object is not a list. However, when you iterate over it, it does return the matching elements.
<syntaxhighlight lang=python>
import re
filterobject = filter(re.compile(<regular expression>).search,list1) # This returns a filter object you can iterate over but not slice like filterobject[0]
newlist = list(filter(re.compile(<regular expression>).search,list1))) # This returns a real list
</syntaxhighlight>

;Select from a list of lists.
<code>newist = [i for i in lists if 'SERACHSTRING' in str(i)]</code>

;Check if some values are in a list (works for sets too)
<code>if [ i for i in alist if i in [value1,value2]]:</code>

;sum(aListWithNumbers)
:Total of all values in the list

==set==
Class of iterable, mutable objects. Objects added to sets are hashed. Therefor:
* Only immutable objects can be added to a set. So not dictionaries or lists
* Sets cannot hold duplicate objects (adding an object again does not change the set).
* Checking if a set holds an object is very fast.

A set is iterable but has no order, therfor:
* You can loop over a set like <code>for a in set:</code>
* You cannot take a [[Python:DataTypes#Slicing | slice]] from a set.

;set1 = set()
:Initialize an empty set

;set1 = set([<values>])
;set1 = {<val1>,<val2>}
:Initialize a set with objects. Note the list-format of <values>.

;set2 = set1.copy()
:Make a copy of set1. <code>set2 = set1</code> will make set2 point to the same set, changing set1 will change set2 (as it is the same object).
;set1.add(2)
:Add the '2' object to set1. You can add only 1 object at a time.

;set1.discard(2)
:Remove the '2' object from set1 (returns None object)

;set1.update(itterable)
:Update the set with the elements in itterable (list, set, )
;unionset = set1.union(set2)
:Combine the sets into a new set (e.g.to remove duplicates)

;intersectset = set1.intersection(set2)
;intersectset = set1 & set2
:Put all elements that exist in both sets into a new set

;diffset = set1 - set2
;diffset = set1.difference(set2)
:diffset will have all elements of set1 that are not in set2

Augmented assignment (update the left hand object instead of creating a new one) are:
;set1 |= set2
:Update set1 with set2
;set1 -= set2
:Removed set2 from set1
;set1 &= set2
:set1 will have the elements that exist in both sets (intersection)
;set1 ^= set2
:set1 will have the elements that exist in only 1 of the sets

==Tuple==
Class of iterable, immutable objects. It acts as an immutable [[#list|list]]. Multiple return values from functions are tuples and e.g. results from database queries are by default returned as tuple.

;tpl1 = ()
:Initialize an empty tuple (pretty useless action)

==Dictionary or dict==
Class of iterable, mutable objects. Dictionary's can be compared to perl hashes. Check the [[Python:JSON]] page too.

;dict1 = {}
:Initialize an empty dictionary.

;dict1 = { <nowiki>column1: value1, column2: value2</nowiki> }
:Initialize dictionary with data

The [[Python#collections|collections defaultdict]] module can be used for dictionairies with automatic key creation.

;WARNING; In a [[Python:DataTypes#Dictionary_or_dict|dictionary]] created with defaultdict a key will be added when you try to read a non-existing lower key.
<syntaxhighlight lang=python>
from collections import defaultdict
adict = defaultdict(lambda: defaultdict())
print(adict[key1][key2]) # This will create adict[key1]
</syntaxhighlight>

;adict = defaultdict(<type>)
:Create a dictionary key of the provided type automatically when it is used (see WARNINIG above). Use this to avoid checking if a key already exists before you populate it. If you do not provide <type> you can put in anything.
;bdict = dict(adict)
:bdict will be an ordinary dict.

;adict = defaultdict(lambda<nowiki>: defaultdict(lambda: defaultdict()))</nowiki>
:Use lambda function to handle multilevel dictionaries
:NOTE: For 2 levels, and 2nd level is a dictionary you can use <code>defaultdict(dict)</code> too.

;if key in dict:
:Test if key exists in dict. <code>if dict[key]:</code> will throw a keyerror if it does not exist.

;dict1.keys()
:Iterable of keys in dict1
;dict1.values()
:Iterable of values
;dict1.items()
:iterable of all key-value pairs in dict1 as [[#Tuple|tuple]]s

;sum(adict.values())
:Total of all values in a dict if all are numbers

;dict2 = dict1.copy()
;dict2 = dict(dict1)
:Make a copy of dict1. dict2 = dict1 does not make a copy dict2 points to the same object as dict1

;dict1.update(dict2)
:Add dict2 to dict1. Duplicate keys are overwritten in dict1.
:Overwriting happens in multilevel dictionary's too obviously, use something like this
<syntaxhighlight lang=python>
for key in bdict:
for key2 in bdict[key]:
adict[key][key2] = bdict[key][key2]
</syntaxhighlight>

;<code>dict1.pop(key,None)</code>
:Remove key from dict1, return dict1[key] if successful, None if key does not exist in dict1.

Code example recursively search a key:
<syntaxhighlight lang=python>
def search_dict(data,skey=None):
result = None
if skey in data:
result = data[skey]
else:
for key in data:
if isinstance(data[key],dict):
result = search_dict(data[key],skey)
return result
</syntaxhighlight>

Create a list of dict values where the key matches a string (List comprehension for dicts)
<syntaxhighlight lang=python>
[value for key, value in d.items() if 'searchstring' in key]
[value for key, value in d.items() if re.search(pattern,key)]
[value for key, value in d.items() if key == 'keyname']
# List of keys that match
[key for key in d.keys() if key == 'keyname']
</syntaxhighlight>
This very non-intuitive syntax is the same as:
<syntaxhighlight lang=python>
list1 = list[]
for key,value in d.items():
if 'searchstring' in key():
list1.append(value)
return list1
</syntaxhighlight>

;<code>sorted(dict1.items(), key=lambda x: x[1], reverse=False)</code>
:Return a list of tuples (key,value) sorted on value
;<code>sorted(adict.keys(), key=adict.get)</code>
:Return a list of keys sorted on values

;intersectionkeys = dicta.keys() & dictb.keys()
;intersectionkeys = set(dicta.keys()) & set(dictb.keys())
:As dictionary keys are a set-like objects in python3 you can make intersctions like.
:Second format is for python2

==None==
The <code>None</code> object is returned e.g. if nothing is found in a re.search. The <code>None</code> object is '''not''' an empty string

==Range==
Constructor of immutable sequences of integers. Use with list, set, tuple to create the desired object, or for loops.

;range(start,stop,step)
:Generic format. If you leave out step, step = 1. If only 1 parameter is provided, it is the stop number, start = 0, step = 1

<syntaxhighlight lang=python>
for i in range (2,5,1):
print(i)
2
3
4
</syntaxhighlight>

;alist = [*range(5)]
:Unpack the range immediately (that is what * does). alist = [0, 1, 2, 3, 4]

Module [[Python#itertools|itertools]] can be used to loop integers to infinity
<syntaxhighlight lang=python>
import itertools

for i in itertools.count(step=15):
if breakcondition:
break
print(i)
</syntaxhighlight>

==DateTime==

<syntaxhighlight lang=python>
from datetime import datetime
timestamp = datetime.now().strftime("%y%m%d-%H%M%S")
</syntaxhighlight>
Other formats:
:%s - Seconds since [https://www.epochconverter.com/ epoc] (January 1, 1970 00:00:00)
:%f - Nanoseconds or Milliseconds depends on what your system supports
All formats can be found at W3schools[https://www.w3schools.com/python/gloss_python_date_format_codes.asp]
;datetime.datetime(year, month, day, hour, minute, second).strftime('%s')
:Date to [https://www.epochconverter.com/ epoc]; hour, minute and second are optional.

;Get an unique integer. Time is cheaper than datetime.
<syntaxhighlight lang=python>
import time as t
# Use UTC-time 1 tick per second
nonce = int(t.mktime(t.gmtime()))
# This has greater precision (nanoseconds) but is in local time, not unique when the clock is set back.
nonce = int(t.time()*10000)
# I use datetime as that is loaded most time anyhow for other timestamps
from datetime import datetime
timestamp = datetime.utcnow().strftime("%s%f")
</syntaxhighlight>

;Convert datetime strings to a datetime object. The format indicates where the elements can be found (as in unix date formats ('%Y%m%d_%H%M'))
<syntaxhighlight lang=python>
from datetime import datetime
dt = datetime.strptime(<datetimestring>, <format>)
</syntaxhighlight>
;For unixtime however you can not use strptime with %s
<syntaxhighlight lang=python>
dt = datetime.fromtimestamp(<unixtime>)
</syntaxhighlight>
;To convert to a date object
<syntaxhighlight lang=python>
from datetime import datetime
dateobj = datetime.strptime(<datetimestring>, <format>).date()
</syntaxhighlight>

;The parser is pretty smart, but it can be confused for day numbers under 13. It accepts e.g. <nowiki>1-jan-1991 12:30</nowiki> and YYYYMMDD-HHMM format.
<syntaxhighlight lang=python>
from dateutil.parser import parse
datetimeobject = parse(datestring)
</syntaxhighlight>

;Calculations (7 days before now)
<syntaxhighlight lang=python>
from datetime import date
from datetime import timedelta
newdate = date.today() - timedelta(days=7)
</syntaxhighlight>

<syntaxhighlight lang=python>
timddiff = datetimeobject - datetimeobject2
print(timediff.days)
print(timediff.seconds) # max 1 day
print(timediff.microseconds) # max 1 second
</syntaxhighlight>

=Slicing=
You can address all iterable datatypes partly or in a difference sequence.

;object<nowiki>[b:e:s]</nowiki>
:Generic format where b=Begin (counting starts at 0), e=End, s=Stepsize (negative stepsize starts counting at the end)

Examples:
<syntaxhighlight lang=python>
'last element'[-1]
'All elements except the last'[:-1]
'All elements in reversed order'[::-1]

'All elements from the second'[1:]
'Second until 5th element (element 1,2,3 and 4)'[1:5]
'Elements in reversed order (element 5,4,3 and 2)'[5:1:-1]
'Element 1,3 and 5'[1:6:2]
</syntaxhighlight>

Nginx

2026-05-12T19:12:52Z

Hdridder:

[[category:Webdesign]]
[https://nginx.org/ Nginx] is a webserver working very well on small systems

;nginx -t
:Test configuration

;nginx -V
:Print version, compiler version and configured parameters

=PHP support=
To enable php suppport in ./sites-available/xxx put this
location ~ \.php$ {
#NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
fastcgi_intercept_errors on;
fastcgi_param HTTPS on;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;

}

=Tips=
# Put al customizations in your site-definitions (in ./sites-available) or in .conf files in the ./conf.d subdir. This make it easier to migrate your configuration to another server.

==Force clients to use https (HSTS)==
Add to ./conf.d/myconfig.conf
:<code>add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;</code>

Wipe browser history for the HSTS websites if HSTS settings were disabled but the max-age has not passed yet for your browser (31536000 seconds is 1 year).

==Other recommenations==
In the http section (./conf.d/myconfig.conf) put
<syntaxhighlight lang=text>

# For security hide the server version
server_tokens off;
ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSL, TLSv1 and TLSv1.1

# For performance
client_body_timeout 12;
client_header_timeout 12;
keepalive_timeout 15;
send_timeout 10;

client_body_buffer_size 10K;
client_header_buffer_size 1k;
client_max_body_size 8m;
large_client_header_buffers 4 4k;
</syntaxhighlight>

==Block unwanted agents==
To block specific user agents, like GPTbot that can be really aggressive.

In the http section (./conf.d/myconfig.conf) add
<syntaxhighlight lang=text>
map $http_user_agent $blocked_user_agent {
default 0;
~*meta-externalagent 1;
~*GPTBot 1;
~*Amazonbot 1;
}
</syntaxhighlight>

Then in each server that needs the blocking add
<syntaxhighlight lang=text>
if ($blocked_user_agent) {
return 403;
}
</syntaxhighlight>

=Issues and fixes=
;could not build server_names_hash, you should increase server_names_hash_bucket_size<nowiki>:</nowiki> 32
:Due to many servernames in a server definition. In a .conf file in /etc/nginx/conf.d put a line like
:<code>server_names_hash_bucket_size 64;</code>

;Using listen <nowiki>[::]:80</nowiki> ipv6only=on;
:The ipv6only parameter can be set only once. If you have several virtual host you should not use it.

Python:Files

2026-05-03T10:16:12Z

Hdridder: /* Filehandling and metadata */

[[Category:Python]]

=Basics=
;basename = path.split('/')[-1]
:Get the filename from a path
;dirname = re.sub('/[^/]+/$', '/', path)
:Get the directory name from a path
;glob.glob(filespec)
:Return a [[Python:DataTypes#list|list]] of files matching 'filespec'.
Code example:
<syntaxhighlight lang=python>
import glob
files = glob.glob(filespec)
</syntaxhighlight>

;os.path.isfile(filename)
:Boolean for file existence

;fh = open (filename,"r")
:open filename for read and return the filehandle fh. Use w for write, a for append.
;fh.write(line)
:Write line to file opened on filehandle f1.
:NOTE: you have to make sure the line-end is in the line too.
;fh.close()
:Close the file for filehandle fh.
;sys.stdout.write(<string>)
:Write to standard output

Code example:
<syntaxhighlight lang=python>
import os
if os.path.isfile(filename):
f1 = open (filename,"r")
for line in f1:
<codeblock>
f1.close()
</syntaxhighlight>
Or 'Easier to Ask for Forgiveness than Permission' (EAFP):
<syntaxhighlight lang=python>
try:
fh = open (filename,"r")
except:
print('ERROR: {} cannot be opened'.format(filename))
logging.error('ERROR: {} cannot be opened'.format(filename))
else:
<other code>
</syntaxhighlight>

;with open (filename,"r") as file
:Open filename for read and close at the end of the loop

Code example:
<syntaxhighlight lang=python>
with open (filename,"r") as file:
for line in file:
<codeblock>
</syntaxhighlight>

==Other ways of reading==
;f1.read(size)
:Return 'size' bytes from the file as [[Python:Strings|string]]. If size is omitted or 0 the entire file is returned.

;f1.readlines()
;list(f1)
:Return all lines from file as [[Python:DataTypes#list|list]].

;fileinput.input()
:Read through all files specified on the commandline.
:If there are no files on the commandline read standard input
:You can pass other arguments too but you have to remove them from sys.argv before you start reading fileinput
;fileinput.filename()
:The name of the we are working on

<syntaxhighlight lang=python>
import fileinput
import sys

otherarg = sys.argv.pop() # other argument is the last on the commandline

for line in fileinput.input():
<codeblock>
</syntaxhighlight>

=Filehandling and metadata=
;os.unlink(filename)
:Remove file or symbolic link

;shutil.copy(source, destination)
:Copy file from source to destination, destination can be both a file or a directory
;shutil.copyfile(source, destination)
:Copy file from source to destination, both as full path
;shutil.move(source, destination)
:Move file from source to destination, destination can be both a file or a directory

;statinfo = os.stat(filename)
:Get file metadata like:
:<code>posix.stat_result(st_mode=33204, st_ino=3069488, st_dev=21L, st_nlink=1, st_uid=999, st_gid=999, st_size=37078, st_atime=4939053720, st_mtime=3939053719, st_ctime=2939053719)</code>
:<code>statinfo.st_size</code> has the filesize in bytes.

;Walking a directory tree and fetching file information
<syntaxhighlight lang=python>
def do_dir(directory):
with os.scandir(directory) as it:
for entry in it:
if not entry.name.startswith('.'):
if entry.is_file():
filepath = entry.path
inode = entry.inode()
ctime = entry.stat().st_ctime # see statinfo for other data
elif entry.is_dir():
do_dir(entry)
</syntaxhighlight>

=Archives=

==Read an archive==
Read a file in a tar archive into a list of lines regardless the compression used (not zip).
<syntaxhighlight lang=python>
import tarfile
tar = tarfile.open(<tarfile>,'r')
for member in tar.getmembers():
print(member.name)
filelist = tar.extractfile(member)
</syntaxhighlight>

==Copy files from 1 archive to another==
<syntaxhighlight lang=python>
#!/usr/bin/env python3
import tarfile

filenames = {<(part of) filename to copy>, <(part of) filename to copy>}
oldtar = tarfile.open('tar1.tar',"r")
newtar = tarfile.open('tar2.tar',"w")
for member in oldtar.getmembers():
done = 0
for filename in filenames:
if filename in member.name:
try:
newtar.addfile(member, oldtar.extractfile(member.name))
done = 1
except OSError as exception:
print(f"{member.name} has error {exception}")
done = 2

if done == 1:
print(f"{member.name} Added")
elif done == 0:
print(f"{member.name} Skipped")
newtar.close()
oldtar.close()
</syntaxhighlight>

=Zip files=
Check [https://pymotw.com/2/zipfile/ this page].

==Read a zip-file==
<syntaxhighlight lang=python>
import zipfile

z = zipfile.ZipFile(zipile)
for file in z.namelist():
print(file)

data = z.read(<zipped-filename>)
</syntaxhighlight>

==Create a zip-file==
<syntaxhighlight lang=python>
import zipfile,zlib

zipname = filename+'.zip'
zfile = zipfile.ZipFile(zipname, mode='w')
if zfile:
zfile.write(filename, compress_type=zipfile.ZIP_DEFLATED)
</syntaxhighlight>

=[[XLS|Excel files]]=
==Reading==
Excel-files is basically a zip-file with some specific content and they can be handled like that. [[Pandas#Reading_Data]] has a build in ability to read excel into a dataframe. Whenever possible use that, else check [[XLS]].

==Writing==
Below writes a list of lists to excel
<syntaxhighlight lang=python>
import xlsxwriter
import json

def main():
workbook = xlsxwriter.Workbook(excelfilename, {'nan_inf_to_errors': True})

header = ['Column1', 'Column2']

writeworkbook(workbook, worksheetname, alist, header)

workbook.close()
return

def writeworkbook(workbook, worksheetname, outlist, header):
worksheet = workbook.add_worksheet(worksheetname)
columnwidths = {}

columnno = 0
for column in header:
columnwidths[columnno] = len(column)
columnno += 1

for row in outlist:
columnno = 0
for column in row:
try:
columnwidths[columnno] = max(columnwidths[columnno], len(str(column)))
except KeyError:
# The rows may have variable length
columnwidths[columnno] = len(str(column))
columnno += 1

for columnno in columnwidths:
worksheet.set_column(columnno, columnno, columnwidths[columnno] + columnwidths[columnno] * 0.1)

wsindex = 0
worksheet.write_row(wsindex, 0, header)
for row in outlist:
columnno = 0
for column in row:
if type(column) in (list, tuple, dict):
row[columnno] = str(column)
columnno += 1
wsindex += 1
worksheet.write_row(wsindex, 0, row)

return

main()
</syntaxhighlight>

=Read from standard input and keyboard=

Read from standard input
<syntaxhighlight lang=python>
import sys

for line in sys.stdin:
<codeblock>
</syntaxhighlight>

Prompt and read from keyboard into a
<syntaxhighlight lang=python>
a = input("Prompt: ")
</syntaxhighlight>

In python2
<syntaxhighlight lang=python>
a = raw_input("Prompt: ")
</syntaxhighlight>

=Read a csv=
This code read all files matching the specification and return the content as a list of dicts that have the fieldnames as keys. Fieldnames must be on the first line of the file an must be unique.
NOTE: This code cannot handle value's that contain the separator. The line will be split on all separator occurrences. Use [[Pandas]] or a specific csv-reader module if you need this.

<syntaxhighlight lang=python>
def csv2dict(filespec, separator=','):
'''Convert a csv-file to a list of dicts'''
outfile = []
filedir = glob.glob(filespec)
for filename in filedir:
try:
fh = open(filename, "r")
except:
print('{} cannot be opened'.format(filename))
else:
filelist = [line.strip().split(separator) for line in fh]
fh.close()
header = filelist.pop(0)
fieldnames = set(header)
if len(header) != len(fieldnames):
print('ERROR: Fieldnames in {} are not unique'.format(filename))
else:
numfields = len(header)
linecount = 0
for line in filelist:
linecount += 1
linedict = {}
count = 0
for field in line:
linedict[header[count]] = field
count += 1
if count > numfields - 1:
break
if count != numfields:
print('ERROR: invalid number of fields in line ' + str(linecount))
outfile.append(linedict)
return (outfile)</syntaxhighlight>

=Read xml=
Module and code examples [[Python:XML]]

Python:Functions

2026-04-28T09:50:39Z

Hdridder: /* Build in functions */

[[category:Python]]

;sys._getframe( ).f_code.co_name
:Return name of current function

;eval(functionnameasstring)()
:Convert a function name as string to a function call
:WARNING: Make sure you have control over the string provided, you may call anything this way.

=Passing arguments=

If an argument is a mutable object it is passed by reference. So when you modify the object in the function, the change there in the outside world too.
Immutable objects are passed by value. You cannot change the object, you have to return the new one.

<syntaxhighlight lang='python'>
def main():
aInt = 1
aList = ['a']
bInt = function(aInt, aList)
print(aInt, bInt, aList) # This will print: "1 6 ['a', 'something']"
return

def function (a, b):
# a is a new integer object
# b is a reference to the aList object.
a += 5
b.append('something')
return a

main()
</syntaxhighlight>

Arguments are positional but can be passed by keyword too.

<syntaxhighlight lang='python'>
def main():
outcome = function(par1,par2)
function(par2 = <value>, par1 = <value>)
return

def function (inpar1, inpar2 = <defaultvalue>):
codeblock
return result

main()
</syntaxhighlight>

;def function (*args,**kwargs):
:Take any number of positional arguments and put them in the [[Python:DataTypes#Tuple|tuple]] args
:Take any number of keyword arguments and put them in the [[Python:DataTypes#Dictionary_or_dict|dict]] kwargs

;You can pass a function-name as parameter

<syntaxhighlight lang='python'>
def main():
outcome = function1(function2,inpar2 = <value>)
return

def function1 (functiontocall, inpar2 = <defaultvalue>):
result = functiontocall(inpar2)
return result

def function2 (inpar):
codeblock
return result

main()
</syntaxhighlight>

=Anonymous functions(lambda)=
lambda defines an anonymous function.

Can be used in may places.

Here the dictionary has lambda functions as value. adict['a'](<value>) calls the function and returns True of False.
<syntaxhighlight lang=python>
adict = {
'a': lambda a: 0 < a < 10,
'b': lambda a: 11 < a < 20,
'c': lambda a: 21 < a < 30,
'd': lambda a: 31 < a < 40
}

adict['a'](5) # True
adict['a'](15) # False
</syntaxhighlight>

=Build in functions=

List of all build in functions [https://docs.python.org/3/library/functions.html]

;len(obj1)
:Return the number of elements in obj1 (string, list, tuple, set)

;max(obj1)
:Return the largest item in obj1

;min(obj1)
Return the smallest item in obj1

;round(x,y)
:Round x to y decimals, y defaults to 0
;BEWARE; In python 3x rounding does not work 'As taught in school'[https://stackoverflow.com/questions/10825926/python-3-x-rounding-behavior] but rounds half to even[https://en.wikipedia.org/wiki/Rounding#Round_half_to_even].
Use this to implement 'As taught in school' by overwriting the round() function.

<syntaxhighlight lang=python>
def round(valuein, digits=None):
import decimal
value = str(valuein)
if digits is None:
try:
digits = len(value.split('.')[1])
except IndexError:
digits = 0
formatstring = '1.' + '1' * digits
try:
result = decimal.Decimal(value).quantize(decimal.Decimal(formatstring), rounding=decimal.ROUND_HALF_UP)
except decimal.InvalidOperation:
result = valuein
return result
</syntaxhighlight>

;input([prompt])
:Read and return input from standard input. Display 'prompt' first when provided.

;int(string[,base=x)])
:Convert str to an integer. Base = 2 converts bit-string presentation ('010101').

;str(object)
;repr(object)
:Return a string representation of an object. str for the informal representation (just make it printable), repr for the formal representation (more formating??)

=Boolean functions=
Functions that just return True or False
;isinsantce(var, type)
:If a variable has the wanted type (list,set,int,str,dict)
;isfile(filename)
:If a file exists

;any(<object>)
:If any of the elements is True (any iterateble will do)
;any(x in alist for x in aset)
:If any of the elements of list are in set
;all(<object>)
:True if all elements of <object> are true (have a value)

=Resursive functions=
In resursive functions you should update the objects to return, not create new ones. Use augmented assignment (+=).

=Other often used functions=
;Sleep function
<syntaxhighlight lang=python>
import time
time.sleep(<sec>)
</syntaxhighlight>

Html

2026-04-26T13:18:08Z

Hdridder:

[[Category:Webdesign]]

<h1>Redirect</h1>
;<meta http-equiv="Refresh" content="0; url='https://wiki.deridder.it'" />
:Immediate redirect to this wiki.
<h1>Forms</h1>
;<form action="'.htmlspecialchars($_SERVER['REQUEST_URI']).'" method="post">
:Start a form
:;action
::Page to load when submit is done. This loads the same page as making the request
:;method
::Submit method (GET or POST)
<h2>Input boxes</h2>
;<input type="text" name="varname" value="xxx" size=20 readonly disabled>
:An input box to use in forms
:;name
::Name of the key in $_POST returned
:;value
::Default value
:;readonly
::No input allowed
:;disabled
::Value will not be in $_POST

Pandas

2026-03-24T15:44:18Z

Hdridder: /* Use data */

[[Category:Python]]
Check the [https://pandas.pydata.org/pandas-docs/stable/10min.html 10 minutes to Pandas] too or [https://pythonexamples.org/pandas-examples/ pythonexamples].
;import pandas as pd
:Import the library, we assume this was done on this page

=DataFrame=
Object for tabular data (that is e.g. obtained by read_html).

Each column is a [[Pandas#Series]]

;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}])</nowiki>
:Create a simple dataframe having a range index, 2 rows and 2 columns
;df = <nowiki>pd.DataFrame([{col1: 1,col2: 2},{col1: 1,col2: 2}],[row1,row2])</nowiki>
:Create a dataframe having a list index. The size of index must match the number of rows.

Single columns can be addressed by there name like in <code>df.columnname.unique()</code> to get unique values in a column.

Single or multiple columns can be addressed as list like in <code>df.set_index([column1,column2])</code> to set an index over 2 columns.

;df.copy
:Make a copy of df ( = will point to the same dataframe, like with [[Python:DataTypes#Dictionary or dict|dictionaries]])

;df.join(table2)
:Like SQL-join merge on index
;df.merge(table2,on=<column>)
:Like SQL-join on <column> = df.index
;dfall.merge(table2, on = <column>, how='left', indicator=True)
:Return df joined with table2. Add column _merge with value 'left_only' to records that only exist in df.
;df = dfall[dfall['_merge'] == 'left_only']
:Return dataframe with all records not in table2
;df = pd.concat(dflist)
:Concatenate all dataframes in dflist

==Information==

;df.info()
:Information on dataframe (Index, size, datatypes per column)
;df.size
:Return the number of cells in a dataframe
;df.shape
:Return the number of rows and columns in a dataframe as [[Python:DataTypes#Tuple|tuple]].
;df.describe()
:Return various attributes for numeric columns in the dataframe. This may indicate if there are outlyers.
;df.head(x)
;df.tail(x)
:Return first/last x data rows of df (5 is the default value for x).

;df.index
:Return the table index (class = pandas.core.indexes.base.Index)

;df.columns
:Return the column headers (class = pandas.core.indexes.base.Index)
;df.dtypes
:Return the data-type of the columns
;df.columnname.dtype
:Return the data-type of column columnname

==Modify==

;<nowiki>df.rename(columns={oldname:newname}, inplace=True)</nowiki>
:Redefine the column headers (modifies the dataframe itself, nothing is returned)

;df.set_index(['col1','col2'])
:Return the dataframe with a new index composed of 2 columns
;df.reset_index()
:Return the dataframe with a default index (range)
;df.sort_index(ascending=False)
:Sort the dataframe on the index. 'True' is the default if 'acsending' is not supplied.
;df.index = df.index.map(str)
:For a data-type on the index

;df.drop(index)
;df.drop([indexes])
;df.drop(range(0,3))
:Remove row(s) from a dataframe.
;df.drop(columns=[listofcolumnstodrop]
:Remove columns from a dataframe.

;df.dropna(thresh=x)
:Return only rows that have at least x columns with a none-NA value. If x < 2 the threshold can be dropped.

;df.loc[<index>,['field1','field2']] = [value1,value2]
:Change values for index <index> in a dataframe
;df.fillna(<value>)
;df[[<column1>,<column2>]] = df[[<column1>,<column2>].fillna(<value>)
:Return dataframe with all NA-values (in the selected columns) replaced by <value>

;table.agg(newname=('columname', np.max))
:Add newname to df with the max of columnname. This named aggregations is only supported from version v0.25

;df.transform(<function>)
:Apply a function to an existing column or the entire dataframe.

;df.assign(newcolumn = <expression>)
:Add column newcolumn to the dataframe
:Expression can be a function like "lambda d: d['column'] * <something>"

;df.apply(<function>)
:Apply a function to the dataframe. By default every element of the table.
;<nowiki>df.apply(lambda x: <function>(x) if x.name in [columnname] else x)</nowiki>
:Apply a function to specified colums
;<nowiki>df[columname] = df[columname].apply(lambda x: <function>(x))</nowiki>
:Change a column in the dataframe by applying <function>

;df.replace(<pattern>,<newvalue>,regex=True)
:Replace <pattern> with <newvalue> in the entire dataframe

;df[columna].mask(df[columna] == origvalue , newvalue, inplace=True)
:Change all cells in columa that have origvalue into newvalue (inplace, so in the dataframe itself).

==Select data==
Use .loc, else colomn-names will be considered too (if I understand this[https://stackoverflow.com/questions/38886080/python-pandas-series-why-use-loc] correctly).

;WARNING!!
:at least with .loc if only 1 item is found is it returned as string(/float?) else as dataframe/series.

;df.[[iloc]][<slice>]
:Return the rows indicated by <slice> as pandas.Series
;<nowiki>df.iloc[:,0:3]</nowiki>
;<nowiki>df[[col0,col1,col2]]</nowiki>
:Return 3 columns from the dataframe
;df.filter(regex=<regex>,axis='columns')
:Return all columns which name matches <regexp>. (axis=1)

;df.loc[<index>]
:Select the row(s) with index <index>
;df.loc[<indexname>].<columnname>
;df.loc[<indexname>].[<columnname>]
:Select only 1 column from a row as pandas series. You may convert that to a list or set immediately by using <code>list(df.loc[<indexname>].<columnname>)</code>
;df.loc[0][0]
:Select a cell in a dataframe with no index or header
;<nowiki>df.loc[lambda d: d[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1] == <value> ]</nowiki>
;<nowiki>df.loc[df[column1].notnull()]</nowiki>
:Select all rows where column1 equals <value> or do not have a null-value.
;<nowiki>df.loc[(df[column1] == <value>) | (df[colomn1] == <value2>)]</nowiki>
:Combine conditions with '|' (or) or '&' (and). Use () on each condition!!
;<nowiki>df.loc[datetime.today()-pd.Timedelta(days=7) : datetime.today()]</nowiki>
:Select all rows where the (datetime)-index is between 7 days ago and today.
:NOTE!! The index must be sorted in ascending order[https://stackoverflow.com/q/67298584/2213323].
;df.last('7D')
:Select the last 7 days in the dataframe. For this als the datetime index must be sorted in ascending order

See also the [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.contains.html#pandas.Series.str.contains pandas series pages]
;df[df['fieldname'].str.contains('regexp',regex=True)]
:Select all rows in df where fieldname contens matches regexp. The field must have a string datetype. If you are looking for a simple string you can leave out the regex=True part.
;df[df['fieldname'].gt(value)]
:Select all rows in df where fieldname is bigger than value. The field must have a numeric datetype

;df.filter(regex=<regex>,axis='index')
;df.filter(regex=<regex>,axis='index').<columnname>
;df.filter(regex=<regex>,axis='index').index
:Return all rows for which in index matches <regexp> or get only the column of the matched indexes. (axis=0 ) or the indexname(s).
:The .index returns the matching indexes as a [[Python:DataTypes#List|list]].

;df.sort_values(<columnname>)
;df.sort_values([<column1>,<colunm2>],ascending=(True,False))
:Return the dataframe sorted on the values in the columns. The second form sorts on column1 first and then on column2, column1 ascending, column2 descending
;df.groupby([column1,column2])
:Return a DataFrameGroupBy object, this is not a dataframe
;df.collname.unique()
:Return a [[Numpy]] array with all distinct values in column 'collname'

;Get all rows from df1 that have an index that exists in df2
df = df1[df1.index.isin(df2.index)]

;Or get all intersectioned indices and select by loc:
df = df1.loc[df1.index & df2.index]
df = df1.loc[np.intersect1d(df1.index, df2.index)]
df = df1.loc[df1.index.intersection(df2.index)]

==Use data==
;Print 2 attributes for each row and check for no value
:pd.isna(value) works for Null, NaN (Nod a number) and NaT (not a time) too.
<syntaxhighlight lang=python>
for index, row in df.iterrows():
print(row['ColumnA'],row['ColumnB'])
if ps.isna(row['ColumnA']):
print('Found a NaN value')
</syntaxhighlight>
;Print all values for the first row
<syntaxhighlight lang=python>
for column, row in df.iteritems():
print(column,row[0])
</syntaxhighlight>

;Loop over columns
<syntaxhighlight lang=python>
for column in df:
print(f"Columnname {column}")
</syntaxhighlight>

;df.iloc[0]
:return the values of the first row as pandas series. If there is only 1 column returns the value.

;df.iloc[0][0]
:return the value of the first column in the first row. Columns can also be identified by name

;df[<column>].quantile(q=0.1)
:Return the 10th percentile (10% of the values in <column> is lower than the 10th percentile)

;df[<column].str.extract(r"A regex(p\w+)")
:Return a series (column) of the values selected by [[Regular Expressions]] (p\w+)
;df[<columna>].map(str) + '_' + df[<columnb>].map(str)
:Return a series (column) with the columns concatenated. If not mapped to str an error will be thrown.
;<nowiki>df[<column>].map(str).apply(lambda x: x[-1])</nowiki>
:Return a series (column) with the last character of the <column>

=Series=
Pandas Series [https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html online documentation].<br>
A pandas series is a 1 dimensional array with named keys.<br>
Pandas Series have all kind of methods similar to [[Numpy]] like main, std, min, max,.... In fact Pandas is using numpy to do this.
;s = pd.Series([])
;s = pd.Series([valuelist],[indexlist])
:Initialize a series. If indexlist is omitted the keys are integers starting at 0.
;s[<key>] = <value>
:Assign <value> to the series element with key <key>
:The order in the series is the order in which they are created, NOT the numeric order.
:Elements can be addressed as <code>s[<key>]</code>, <code>s.<key></code> or <code>s[<numkey>]</code>. Where <numkey> is defined by the order the element was created.
:Once you have used named keys in a series you cannot create new elements with a numeric key.
;s.index
:All indexes in the series. Can be sliced to find a particular index.
;s.describe()
:Series statistics

All in 1 example:
<syntaxhighlight lang=python>
import numpy as np
import pandas as pd
s = pd.Series([])
for i in range(50):
s[i] = int(np.random.random() * 100)

for i in s.index:
print(i,s[i])
</syntaxhighlight>

Funny, you can do <code>s[0]</code> but not
<syntaxhighlight lang=python>
for i in s:
print(s[i])
</syntaxhighlight>
To get all values from the series you do:
<syntaxhighlight lang=python>
for v in s:
print(v)
</syntaxhighlight>
To get the indexes too:
<syntaxhighlight lang=python>
for i in s.index:
print(i,s[i])
</syntaxhighlight>

==TimeSeries==
Indexes that contains datetime values are automatically casted to a DatetimeIndex.

;df.resample('5D').mean
:Return the dataframe with the average value of each 5 days.

;<nowiki>df.resample('1H').agg({'ColumnA':'first','ColumnB':'max', 'ColumnC':'min', 'ColumnD':'last' })</nowiki>
:Resample with aggregation specified per column.

;df.resample('5D',origin='end')
:Resample so the last timestamp is used, by default it starts with the first, the last set may be incomplete.

==Datetime functions==
;df[datefield] = pd.to_datetime(df[datefield],format='%d-%m-%Y')
:Convert a string-column with a date into datetime. Format is the input format.
:WARNING: without format to_datetime tries to make up what the format is. This is pretty intelligent but can easily be wrong with day-numbers under 13
;df[datefield] = pd.to_datetime(df[datefield],unit='s')
:Convert a column with [[unix time]] into datetime.
;df[<timefield>] = testdf[<datetime-field>].dt.time
:Return a series (column) with the time-value of a datetime-column

=Reading Data=
;read_html(url)
:Read html tables into a list of [[#DataFrame |dataframes]] (no header, no index)
Example code. The first line in the table is a header, the first column the index (e.g. dates), decimal specifies the decimal point character.
<syntaxhighlight lang=python>
tables = pd.read_html(url,header=0,index_col=0,decimal=<char>)
</syntaxhighlight>

;read_sql(query,cnx,index_columns=[col1,col2])
:Read data from the database opened on cnx (see [[Python:Databases]])
<syntaxhighlight lang=python>
df = pd.read_sql(query,cnx,index_col=['Primarykey'])
</syntaxhighlight>

;read_csv(csvfile)
:Read data from a file with Character Separated Values
;<nowiki>read_csv(csvfile, delimiter=',', usecols=[<columnA>, <columnB>], dtype = {<columnA>: str})</nowiki>
:Read columns <columnA> and <columnB> into a dataframe and force columnA to string type.
;<nowiki>read_csv(csvfile, index_col=False, header=None, names=['columnA', columnB'], na_values = 'None')</nowiki>
:Read from a file with no indexcolumn nor a headerrow, set the colomnnames and handle 'None' and NA value.

==Excel==
;read_excel(xlsfile, sheetname)
:Read data from the sheet with 'sheetname' in a microsoft [[XLS|Excel file]] into a dataframe
;<nowiki>read_excel(xlsfile,sheetname,converters={'columna':str,'columnb':str})</nowiki>
:Force columns to be read as string
;read_excel(xlsfile, sheet_name=None)
:Read all sheets into a dict {sheetname: df,...}. Sheets can be specified by number or name, if multiple, in a list. If only 1 sheet is specified a dataframe is returned, not a dict of dataframes.
<syntaxhighlight lang=python>
dfdict = pd.read_excel(filename, sheet_name=None)
for sheetname, df in dfdict.items():
print(f"Working on sheet {sheetname}")
for index, row in df.iterrows():
print(row)
adict = df.to_dict('records') # Creates a list of dicts [{columname: value,...},...]
print(json.dumps(adict, indent=4))
</syntaxhighlight>

Convert all sheets to CSV files
<syntaxhighlight lang=python>
import sys
import pandas as pd

def main():
for filename in sys.argv[1:]:
try:
dfdict = pd.read_excel(filename, sheet_name=None)
except Exception as exception:
print(f"{filename} cannot be read by pandas.read_excel as {exception}")
else:
for sheetname, df in dfdict.items():
print(f"Working on {filename}, {sheetname}")
if len(dfdict) == 1:
sheetname = ''
else:
sheetname = f"_{sheetname}"
lines = df.to_dict('records')
header = lines[0].keys()
outputfile = filename.replace('.xlsx', f"{sheetname}.csv")
fh = open(outputfile, 'w')
fh.write(','.join(header) + '\n')
for line in lines:
fh.write(','.join(map(str,line.values())) + '\n')
fh.close()
return

main()
</syntaxhighlight>

Disks and filesystems

2026-03-10T10:23:35Z

Hdridder:

[[Category:Linux/Unix]]

;badblocks -v- s /dev/sdb >badblocks.log
:Check a device for bad blocks

;fdisk /dev/sdb # Deprecated
;gdisk /dev/sdb
:Manage disk (GPT) partitions

;df -h
:Show filesystem block usage in human friendly format

;df -i
:Show filesystem inode usage

;df -P
:Report file-system information in parseble format.

;du -shx *
:Report disk usage for all files and directories in the current directory. Do not count files on other filesystems (-x) and do not follow symbolic links (default)

;ls -i <file>
:Show inode of <file>

;stat <file>
:Show about all there is to know about <file>

;getfacl <file>
:Show the Access Control List (ACL) information for a file or directory

;setfacl -m <nowiki>u:<username>:<perm> <file></nowiki>
:Add <perm> for <username> to <file>
;setfacl -d -m <nowiki>u:<username>:<perm> <dir></nowiki>
:-d makes that all files in <dir> will get the <username>:<perm> added to the ACL

;mount -t <fstype> -o <options> <devicefile> <mountpoint>
:Mount a filesystem (fstype and options can be omitted often)

;mount -a
:Mount all filesystems in [[#fstab|/etc/fstab]] with the <code>auto</code> option.
;mount -o loop /path/to/my-iso-image.iso /mnt/iso
:Mount an .iso file

==fstab==
Configuration file to store mounts. Example for NFS mounts:

;<code><host>:<path> <mountpoint> nfs defaults 0 0</code>
:Enable mount by; <code>mount <mountpoint></code>
:Mount the <path> exported by <host> on directory <mountpoint> using the NFS protocol with default setting (see below), do not automatically backup when dump is executed and do not fsck when mounted at boot time.

;<code><host>:<path> <mountpoint> nfs defaults,noauto 0 0</code>
:Same as above but do not mount at boot-time

Defaults (specified by defaults) are for ext3 filesystems: rw,suid,dev,exec,auto,nouser,async (read/write, set userID, dev?, allow execution of executables, mount during boot and when <code>mount -a</code> is done, only root can mount this, do not wait for the host to have written to storage)

See the [https://en.wikipedia.org/wiki/Fstab fstab wikipedia page] for more.

==Performance==
The graphical utility '''Disks''' has a benchmark option.

;dd if=/dev/zero of=./test.tmp bs=1G count=1 oflag=dsync;rm -f ./test.tmp
:Check plain write speed
;dd if=/dev/zero of=./test.tmp bs=512 count=1000 oflag=dsync;rm -f ./test.tmp
:Latency test

On an empty disk you can do:
;dd if=/dev/zero of=/dev/sdb bs=8k count=10k
:Test disk write speed if no filesystem exists

==Misc==
;/dev/null
:The unix black hole. Write always succeeds with no effect at all.

;/dev/zero
:Provides an unlimited amount of null characters (ASCII 0). Can be used for cleaning disks or benchmarking (see #Performance)
:Writing to /dev/zero is the same as writing to /dev/null

Archiving

2026-02-26T13:31:30Z

Hdridder:

Creating archives and compress files.

==zip==
Available on most operating systems
;zip archive.zip file
:Create or update archive.zip, put file in there. If file is already existing in the archive it is replaced.
;zip -r archive.zip path
:Create or update archive.zip, put path and all below in there. Files existing in the archive are replaced.

;unzip archive.zip file
:Extract file from archive.zip and put it in the current directory

;unzip archive.zip path
:Extract path from archive.zip and put it in the current directory. Use quotes if path has wildcards.

;unzip -l archive.zip
:List all files in the archive.

;unzip -l archive.zip path
:List path if it is in the archive.

==tar==
Available on Unix-like operating systems
;tar cvf archive.tar path
:Create archive.tar and put all file under path in it. Be verbose, list what is done (-v)

;tar xvf archive.tar path
:Extract path from archive.tar, put it in the current directory. Be verbose, list what is done (-v)

;tar tf archive.tar path
:List what is in archive.tar.

;tar cvzf archive.tgz path
:Create gzipped archive.tgz and put all file under path in it. Be verbose, list what is done (-v).
:Similar for x and t, the z makes tar work on gzipped archives;

;tar uvf archive.tar --no-recursion --files-from -
:Read filenames from standard input e.g. as found by [[find]] and update the archive.

==gzip==
;gzip <filename>
:Compress file to <filename>.gz

;gzip -c <filename>
:Compress file to standard output. Can be used for log-rotation when the inode of the file should not change as the application keeps it open.
:<code>gzip -c <filename> > <filename>.gz; cp /dev/null <filename></code>

==7zip==
Available on most operating systems. Provides very good compression
;7za a <archivename>.7z <files>
:Add files to archive, create it when needed.
;7za l <archivename>.7z
:List an archive.
;7za e <archivename>.7z <files>
:Extract <files> from archive without path, all if <files> is omitted.
;7za x <archivename>.7z
:Extract <files> from archive with full path, all if <files> is omitted.

Hoofd en Hart

2026-02-13T20:40:37Z

Hdridder: Created page with "Het hoofd bestuurt, het hart regeert."

Het hoofd bestuurt, het hart regeert.

Chat with large language models

2026-01-31T16:21:00Z

Hdridder: /* Get better prompts */

Since 2024 we have Large Language Models (LLM) that are sophisticated enough to have a somewhat reasonable chat with.

Some examples:
* ChatGPT [https://chatgpt.com] by OpenAI, the first public LLM chatbot
* Copilot [https://copilot.microsoft.com/] by Microsoft, based on the OpenAI models
* le Chat [https://https://mistral.ai/products/le-chat] by Mistral, the European alternative (in 2025 not as smart as the othters)
* Gemeni [https://gemini.google.com/] by Google
* Claude [https://claude.ai/] by Anthropic, should be best for coding

An LLM is a implementation of [[Artificial Intelligence]] (AI).

An LLM is trained with a vast amount of text. The model tries to find the most plausible word to present to you given the conversation you are having. Just that, find the next word to say, and at some point decide there is nothing to say until the user says something.

An LLM only knows the things it is trained with and; IT HAS NO CLUE OF WHAT IT MEANS. In below text it may look like chatbots have some intelligence, THEY DON'T. They are just very good at searching the most commonly given answer to your question and formulate that in correct <your language>.

e.g.:
:For chatbots that can generate pictures; Ask it to draw an analogue clock that points to 18:15. It will show you a clock pointing 10:10 (or 13:50) as the vast majority of clock pictures on the web show that time. It may get better over time but in 2025 Copilot offered to make 4 clock that show the time between 18:05 and 18:20 with 5 minutes intervals. It returned 4 clocks showing 10:10.
:You cannot ask a chatbot to count the number of characters in a random word. You may be lucky the word, and the number of characters for it, were in the training data but in general it will return the number that is most times found for counting a word's characters on the web.

=Get better prompts=
Prompts (what you say to the chatbot) should be as specific possible to get the most accurate answer.
You can ask to improve your prompt like (in dutch):
Je bent een expert op het gebied van <chatbot>. Jij helpt mij met het herschrijven van prompts om <chatbot> de meest complete antwoorden voor mij te genereren. Alleen als ik een prompt begin met 'XX:' dan wil ik dat jij de prompt zo herschrijft dat je voldoet aan onderstaande regels:
a) De nieuwe prompt benoemt een expertrol voor <chatbot>
b) De nieuwe prompt geeft de vraagstelling gedetailleerd weer
c) De nieuwe prompt bevat een gedetailleerde opdracht
d) De nieuwe prompt geeft voldoende context
e) De nieuwe prompt is geoptimaliseerd en volledig duidelijk voor <chatbot>
Geef alleen aan of je het hebt begrepen

This prompt will only tell you that is understood you. Next prompts in this chat you start with: "COPILOT-Prompt" will return an optimized prompt.

In CoPilot and chatGPT you can add this prompt improver to you personal setting (without the last command). The chatbot will have this instruction in all future chats.

=Hallucinating=
Chatbots can get lost in the possible answers and start talking rubbish, that is called hallucinating.

All words in the chat are taken into account when formulating answers. That enables the bot to have a (potential) useful conversation with you, however when you change subject it may get confused.

Limit a chat to 1 subject, start a new one when you want to discuss a different subject.

=Bot instructions=
You can tell the bot to be more or less strict in the answers it returns
;Temperature
:Check and add
;forbidden_words_list
:Check and add

Ntp

2026-01-23T15:14:20Z

Hdridder: /* systemd-timesyncd */

[[category:networking]]
Network Time Protocol, synchronize local clock to remote.

To use ntpq the [[iptables|firewall]] should be open for traffic to the localhost.
<syntaxhighlight lang=bash>
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o lo -j ACCEPT
</syntaxhighlight>

When using an NTP-server pool like [https://pool.ntp.org pool.ntp.org] ( 0.<country>.pool.ntp.org etc.) make sure you refresh the peers every now and than. Only when ntp is started the hosts are selected, when they leave the pool ntp will not automatically select another.

;Very secure setinngs in ntp.conf
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

;ntpq
:CLI interface to query the ntp status.
:;peer
::Show peers
:;ass
::Show associations (details of peers)
:;pstat <assid>
::Show status of a association

;ntpdate -q <server>
:Get date from <server> without doing any update

=systemd-timesyncd=
systemd-timesyncd is a lightweight client-only ntp implementation.

By default configuration from DHCP is used. Additional configuration can be added to /etc/systemd/timesyncd.conf
NTP=<space separated list of NTP-servers>
FallbackNTP=<space separated list of NTP-servers>
Only 1 server is used, the server lists are polled one by one until one responds.

;timedatectl show
:Show status of the synchronization
;timedatectl show-timesync
:Detailed status of the synchronization
;timedatectl timesync-status
:More detailed status of the synchronization
;systemctl status systemd-timesyncd
:The status of the daemon
;systemctl restart systemd-timesyncd
:Restart the daemon on configuration changes, reload is not supported.

If you want to use ntp instead of timesyncd e.g. because you want to run a ntp server use:
<syntaxhighlight lang=bash>
systemctl disable systemd-timesyncd.service
apt install ntp
</syntaxhighlight>
The NTP deamon will automatically start with default settings. Adjust things (like the servers) to your likings

Ntp

2026-01-23T14:38:59Z

Hdridder: /* systemd-timesyncd */

[[category:networking]]
Network Time Protocol, synchronize local clock to remote.

To use ntpq the [[iptables|firewall]] should be open for traffic to the localhost.
<syntaxhighlight lang=bash>
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
ip6tables -A INPUT -i lo -j ACCEPT
ip6tables -A OUTPUT -o lo -j ACCEPT
</syntaxhighlight>

When using an NTP-server pool like [https://pool.ntp.org pool.ntp.org] ( 0.<country>.pool.ntp.org etc.) make sure you refresh the peers every now and than. Only when ntp is started the hosts are selected, when they leave the pool ntp will not automatically select another.

;Very secure setinngs in ntp.conf
restrict -4 default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

;ntpq
:CLI interface to query the ntp status.
:;peer
::Show peers
:;ass
::Show associations (details of peers)
:;pstat <assid>
::Show status of a association

;ntpdate -q <server>
:Get date from <server> without doing any update

=systemd-timesyncd=
systemd-timesyncd is a lightweight client-only ntp implementation.

By default configuration from DHCP is used. Additional configuration can be added to /etc/systemd/timesyncd.conf
NTP=<space separated list of NTP-servers>
FallbackNTP=<space separated list of NTP-servers>
Only 1 server is used, the server lists are polled one by one until one responds.

;timedatectl timesync-status
:Detailed status of systemd-timesyncd
;systemctl status systemd-timesyncd
:The status of the daemon
;systemctl restart systemd-timesyncd
:Restart the daemon on configuration changes, reload is not supported.

If you want to use ntp instead of timesyncd e.g. because you want to run a ntp server use:
<syntaxhighlight lang=bash>
systemctl disable systemd-timesyncd.service
apt install ntp
</syntaxhighlight>
The NTP deamon will automatically start with default settings. Adjust things (like the servers) to your likings

Nftables

2026-01-18T20:24:40Z

Hdridder: /* Commands */

[[category:networking]]
[[category:Linux/Unix]]
nftables is the successor of [[iptables]] for managing firewall rules.

At first glance [https://wiki.debian.org/nftables] it looks a bit like a [[iptables#ufw|ufw]]

{| class="wikitable" style="width:35%"
|+ Address families
! Family
! Remark
|-
| ip
| IPv4 This is the default if no family is specified
|-
| ip6
| IPv6
|-
| inet
| IPv4 + IPv6
|-
| arp
| arp
|-
| bridge
| bridge
|}

=Install and activate=
Debian comes with nftables installed since Buster.
;apt install nftables
:Install if needed
;systemctl enable nftables.service
:Create the service
;systemctl start nftables
;service nftables start
:Start the service
;systemctl status nftables
;service nftables status
:Show status

=Confiuration=
The configuration is in /etc/nftables.conf

=Commands=
;nft list ruleset
:Show the current ruleset in the format you can load with nft -f (see below)
;nft list table <family> filter
:Show the current filters for <family>.
;nft -f <filename>
:Load rules from <filename>

Dhcp

2026-01-17T17:28:43Z

Hdridder: /* KEA */

[[Category:networking]]
[[Category:Linux/Unix]]
Dynamic Host Configuration Protocol

Provide IP addresses for your network from a central server.
==IPv6==

[[IPv6]] addresses can be advertised by the radvd (often on your ISP's router). These are based on the MAC address of the client or pseudo randomly generated. For more control over the addresses you can run a dhcp server. When the [https://www.isc.org/dhcp/ ISC dhcp server] is used you need separate instances for the IPv4 and the IPv6 server.

=KEA=
KEA[https://kb.isc.org/docs/aa-01615] is the successor of the isc-dhcp-server. The [[IPv4]] part works fine. [[IPv6]] appears to be a lot more difficult (like it was for isc-dhcp-server)
;apt install kea
:Install all packages for the KEA-server

=isc-dhcp-server=
Sample configuration file (/etc/dhcp/dhcpd6.conf).
<syntaxhighlight lang=bash>
log-facility local7;
default-lease-time 86400;
max-lease-time 604800;

subnet6 your:site:ipv6:prefix::/64 {
# range6 your:ipv6:prefix:address:1::f000 your:ipv6:prefix:address:1::ffff;

option dhcp6.domain-search "local.domain.tld";
option dhcp6.name-servers <name-server-address>;
option ntp-servers ntp.domain.tld;

host host1name {
host-identifier option dhcp6.client-id 00:a4:00:01:45:26:4d:8e:98:8e:76:d1:09:32;
fixed-address6 <full ipv6-address>;
}
</syntaxhighlight>
Hosts for fixed addresses are found by their DUID. Only way I know to find it on the server is from the logfile (/var/log/syslog). On the client it is in var/lib/dhcpcd/duid

The firewall on the client needs to be open on the Link Local address (in the FE80::/10) range on port 546

==IPv4==
Sample configuration file (/etc/dhcp/dhcpd.conf).
<syntaxhighlight lang=bash>

#
ddns-update-style none;

# option definitions common to all supported networks.
option domain-name "domain.tld";
option domain-name-servers 192.168.1.2, 208.67.222.222;
option routers 192.168.1.254;
option broadcast-address 192.168.1.255;
option ntp-servers 192.168.1.4;

# 1 day (fine for home networks, e.g. hotspots should have far shorter lease times)
default-lease-time 86400;
# 1 week
max-lease-time 604800;

authoritative;

log-facility local7;

# Known hosts get fixed IP address defined in DNS-server
group {
deny unknown-clients;
# One week, these are sort of fixed addresses.
default-lease-time 604800;

host host1name {
hardware ethernet ff:ff:ff:ff:ff:ff;
fixed-address host1name.domain.tld;
}
host host2name {
hardware ethernet ff:ff:ff:ff:ff:ff;
fixed-address host2name.domain.tld;
}
}
# All other get an address from the range
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.199;
}
</syntaxhighlight>

The leases are kept track of in /var/lib/dhcp/dhcpd.leases
An ordinary entry looks like:
tstp and cltt are for the failover protocol, most other entries are self-explaining I think.
<syntaxhighlight lang=bash>
lease 192.168.1.133 {
starts 6 2021/03/18 11:22:00;
ends 0 2021/03/19 11:22:00;
tstp 0 2021/03/19 11:22:00;
cltt 6 2021/03/19 11:22:00;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet ff:ff:ff:ff:ff:ff;
client-hostname "host3name";
}
</syntaxhighlight>

=DHCP clients=
;dhcpcd -k <interface>
:Release lease
;dhcpcd -n <interface>
:Renew lease

=Traffic check=
;tcpdump -i eth0 port 547
:Check on server side
;tcpdump -i eth0 port 546
:Check on client side

Perl

2026-01-15T13:18:23Z

Hdridder:

[[Category:Programming]]
=Warnings=
===Put a ; at the end of each line===
This is still what I forget most.

===autovivification===
:If testing for existence of a hash key beware that all intermediate keys will be created by the test. In below example <code>$hash1($key1}</code> will be created if it does not exist yet.
<syntaxhighlight lang=Perl>
if (exists $hash1{$key1}{$key2) {
...
} else {
...
}
)
</syntaxhighlight>
=Things that do work=

Searching and matching is explained in [[Regular Expressions]].

;perl -e '<commands>'
:Execute <commnands>

;Print out the contents of $variable in a structured format:
<syntaxhighlight lang=Perl>
use Data::Dumper;
print Dumper($variable);
</syntaxhighlight>

if-then-else:
<syntaxhighlight lang=Perl>
if < expr > {
<codeblock>
} elsif < expr ) {
<codeblock>
} else {
<codeblock>
}
</syntaxhighlight>

;$var += <value>
:Add <value> to $var (works for -, *, / too)

;$string .= $addstring
:Concatenate $string and $addstring

;$string =~ s/<regexp>/<newvalue>/[g]
:Substitute <regexp> with <newvalue> in $string. The g modifier makes the all occurrences of <regexp> are substituted.

;$newstring = substr($string,start,length)
:Return substring of $string

;sprintf(format,$string)
:Return formatted string. format is e.g. "%.3f". Check [[Python:Strings#Advanced]] for all formats.

;$exitcode = system("a command")
:Executed the command in a subprocess and return the exitcode

;$output = `a command`
:Execute the command and return all command output

;sleep(x)
:Sleep for x seconds

;$|++
:Set standard output to unbuffered, every is put on screen immediately.

;$username = getpwuid($<)
:Get the username of the one executing the script. $< is the uid of the current user.

==Hashes==
;%newhash = (%oldhas1,%oldhash3)
;$newhashref = {$oldhasref1,$oldhashref3}
:Merge 2 hashes, works if all keys are unique

==Arrays==
;arr1 = split(<sep>,$string)
:Split a string in to an array on <sep>

;a = arr1[0]
:First element in arr1

;a = arr1[-1]
:Last element in arr1

;if ( grep(/^<value>$/,@arr1) )
:Check if <value> exist in arr1 (Note the ^ and $ for an exact match

==Files==
;open (FH, "< filename") || die("Open failed for filename")
:Open filename for reading on filehandle FH. Exit the program when the open failed

;@array = <FH>
:Read the file opened on FH into @array (1 element per line

;while ( line = (<FH>) )
:Read the file opened on FH line by line

==Flow control==
The usual works (if-then-else, for , foreach)

;||die("message")
:|| causes the statement only to be executed if the previous failed. Stop the program and write messaged to standard error.

;|| do {codeblock};
:Like die but not exiting the program, use to do additional actions on the failure of the previous command

==Compare==
;==, !=, >=, >, <, etc
:Compare numeric data
;eq, ne, ge, gt, lt, etc
:Compare strings

Python:Databases

2026-01-14T13:58:14Z

Hdridder: /* mariadb */

[[Category:Python]]
= MySQL Connector =

See below for an example script

;import mysql.connector
:Load module for mysql

;cnx = mysql.connector.connect(**cnxconfig)
:Open MySQL database as defined in the cnxconfig dictionary. See example below. The ** passes the dictionary unpacked to a function.

;import cx_Oracle
:Load module for oracle

;cnx = cx_Oracle.connect(user=dbuser, password=dbpasswd, dsn=dbtns_alias)
:Open Oracle database.

;cnx.commit()
:Commit all open transactions to the database

;cursor = cnx.cursor()
:Create a cursor that returns rows as [[Python:DataTypes#Tuple | tuple]]

;cursor = cnx.cursor(dictionary=True)
:Create a cursor that returns rows as [[Python:DataTypes#Dictionary_or_dict | dictionary]] { column1: value1, column2: value2 }

;cursor = cnx.cursor(buffered=True)
:Create a cursor that retrieves the complete result at once so you can check rowcount immediately. Beware of overloading your memory if you expect large results.

;cursor.execute(query)
:Execute 'query'. Traverse the cursor to fetch the data. The cursor returns a tuple for each row by default.<br>
:;cursor.rowcount
::The number of rows fetched (SELECT) or affected (INSERT,UPDATE) by this query.
:;cursor.column_names
::The columnnames

;cursor.fetchone()
:Return the next row from the cursor result.

;cursor.fetchmany(size=2)
:Return the first two rows of the result in a list.

;cursor.fetchall()
:Return all (remaining) rows of the result in a list.

<syntaxhighlight lang=python>
for row in cursor:
print(row)
for column in row:
print(column)
</syntaxhighlight>

;cursor.close()
:Close the cursor
;cnx.close()
:Close the database connection

Code example showing all:
<syntaxhighlight lang=python>
import mysql.connector

cnxconfig = {
'host': 'hostname',
'port': 3306,
'user': 'username',
'password': '',
'database': 'DBname'
}

#cnx = mysql.connector.connect(host='hostname', port=3306, user='username', database='DBname')
cnx = mysql.connector.connect(**cnxconfig)

if cnx:
print("Connection, continue")
else:
print("No connection, exit")
exit()

print("Tuple cursor")
cursor = cnx.cursor()

query = ("SELECT * FROM table")

cursor.execute(query)
for columname in cursor.column_names:
print(columname)
for row in cursor:
print(row)
for column in row:
print(column)

print("Dictionary cursor")
cursor = cnx.cursor(dictionary=True)
cursor.execute(query)
for row in cursor:
for columname in row.keys():
print("{0:25s}: {1:}".format(columname, row[columname]))

cursor.close()
cnx.close()
</syntaxhighlight>

=mariadb=
The mariadb module[https://mariadb.com/docs/connectors/mariadb-connector-python][https://mariadb-corporation.github.io/mariadb-connector-python/] works almost the same as the mysql.connecctor above.

<syntaxhighlight lang=python>
import mariadb

# Connection parameters
config = {
'host': 'xxx',
'port': 3306,
'user': 'user',
'password': 'password',
'database': 'db'
}

# Connect to MariaDB
conn = mariadb.connect(**config)
cursor = conn.cursor()
cursor.execute(<query>)
for row in cursor:
print(row)
</syntaxhighlight>

The rows are tuples with a counter (id) as first element. <code>'dictionary': True</code> in the connection config should return dictionaries
;cursor.description
:Spec of the tuples in cursor
;cursor.metadata['field']
:The fields in the cursor

= ODBC connections =

<syntaxhighlight lang=python>
import pyodbc

dsn = 'DSN=<name of ODBC-DNS>'

cnx = pyodbc.connect(dsn,autocommit=True)
cursor = cnx.cursor()

query = 'show databases'

result = cursor.execute(query)
columnnames = [e[0] for e in cursor.description]
for row in result:
field1 = row[0]

</syntaxhighlight>

Works with [[Pandas#Reading Data|pandas.read_sql]] too.

= MySqlAlchemy =

For more information check the sqlalchemy.org basic usage page[https://docs.sqlalchemy.org/en/11/core/connections.html#basic-usage].

Very basic example for mysql with the pymysql module
<syntaxhighlight lang=python>
import pymysql
import sqlalchemy

dbconnect = 'mysql+pymysql://user:password@databaseserver/databasename'

engine = sqlalchemy.create_engine(dbconnect)

query = 'select something from table'
result = engine.execute(query)
for row in result:
field1 = row[0]

</syntaxhighlight>

Python:Databases

2026-01-14T13:55:45Z

Hdridder: /* mariadb */

[[Category:Python]]
= MySQL Connector =

See below for an example script

;import mysql.connector
:Load module for mysql

;cnx = mysql.connector.connect(**cnxconfig)
:Open MySQL database as defined in the cnxconfig dictionary. See example below. The ** passes the dictionary unpacked to a function.

;import cx_Oracle
:Load module for oracle

;cnx = cx_Oracle.connect(user=dbuser, password=dbpasswd, dsn=dbtns_alias)
:Open Oracle database.

;cnx.commit()
:Commit all open transactions to the database

;cursor = cnx.cursor()
:Create a cursor that returns rows as [[Python:DataTypes#Tuple | tuple]]

;cursor = cnx.cursor(dictionary=True)
:Create a cursor that returns rows as [[Python:DataTypes#Dictionary_or_dict | dictionary]] { column1: value1, column2: value2 }

;cursor = cnx.cursor(buffered=True)
:Create a cursor that retrieves the complete result at once so you can check rowcount immediately. Beware of overloading your memory if you expect large results.

;cursor.execute(query)
:Execute 'query'. Traverse the cursor to fetch the data. The cursor returns a tuple for each row by default.<br>
:;cursor.rowcount
::The number of rows fetched (SELECT) or affected (INSERT,UPDATE) by this query.
:;cursor.column_names
::The columnnames

;cursor.fetchone()
:Return the next row from the cursor result.

;cursor.fetchmany(size=2)
:Return the first two rows of the result in a list.

;cursor.fetchall()
:Return all (remaining) rows of the result in a list.

<syntaxhighlight lang=python>
for row in cursor:
print(row)
for column in row:
print(column)
</syntaxhighlight>

;cursor.close()
:Close the cursor
;cnx.close()
:Close the database connection

Code example showing all:
<syntaxhighlight lang=python>
import mysql.connector

cnxconfig = {
'host': 'hostname',
'port': 3306,
'user': 'username',
'password': '',
'database': 'DBname'
}

#cnx = mysql.connector.connect(host='hostname', port=3306, user='username', database='DBname')
cnx = mysql.connector.connect(**cnxconfig)

if cnx:
print("Connection, continue")
else:
print("No connection, exit")
exit()

print("Tuple cursor")
cursor = cnx.cursor()

query = ("SELECT * FROM table")

cursor.execute(query)
for columname in cursor.column_names:
print(columname)
for row in cursor:
print(row)
for column in row:
print(column)

print("Dictionary cursor")
cursor = cnx.cursor(dictionary=True)
cursor.execute(query)
for row in cursor:
for columname in row.keys():
print("{0:25s}: {1:}".format(columname, row[columname]))

cursor.close()
cnx.close()
</syntaxhighlight>

=mariadb=
The mariadb module[https://mariadb.com/docs/connectors/mariadb-connector-python] works almost the same as the mysql.connecctor above.

<syntaxhighlight lang=python>
import mariadb

# Connection parameters
config = {
'host': 'xxx',
'port': 3306,
'user': 'user',
'password': 'password',
'database': 'db'
}

# Connect to MariaDB
conn = mariadb.connect(**config)
cursor = conn.cursor()
cursor.execute(<query>)
for row in cursor:
print(row)
</syntaxhighlight>

The rows are tuples with a counter (id) as first element. <code>'dictionary': True</code> in the connection config should return dictionaries
;cursor.description
:Spec of the tuples in cursor
;cursor.metadata['field']
:The fields in the cursor

= ODBC connections =

<syntaxhighlight lang=python>
import pyodbc

dsn = 'DSN=<name of ODBC-DNS>'

cnx = pyodbc.connect(dsn,autocommit=True)
cursor = cnx.cursor()

query = 'show databases'

result = cursor.execute(query)
columnnames = [e[0] for e in cursor.description]
for row in result:
field1 = row[0]

</syntaxhighlight>

Works with [[Pandas#Reading Data|pandas.read_sql]] too.

= MySqlAlchemy =

For more information check the sqlalchemy.org basic usage page[https://docs.sqlalchemy.org/en/11/core/connections.html#basic-usage].

Very basic example for mysql with the pymysql module
<syntaxhighlight lang=python>
import pymysql
import sqlalchemy

dbconnect = 'mysql+pymysql://user:password@databaseserver/databasename'

engine = sqlalchemy.create_engine(dbconnect)

query = 'select something from table'
result = engine.execute(query)
for row in result:
field1 = row[0]

</syntaxhighlight>

Python:Databases

2026-01-14T13:53:32Z

Hdridder:

[[Category:Python]]
= MySQL Connector =

See below for an example script

;import mysql.connector
:Load module for mysql

;cnx = mysql.connector.connect(**cnxconfig)
:Open MySQL database as defined in the cnxconfig dictionary. See example below. The ** passes the dictionary unpacked to a function.

;import cx_Oracle
:Load module for oracle

;cnx = cx_Oracle.connect(user=dbuser, password=dbpasswd, dsn=dbtns_alias)
:Open Oracle database.

;cnx.commit()
:Commit all open transactions to the database

;cursor = cnx.cursor()
:Create a cursor that returns rows as [[Python:DataTypes#Tuple | tuple]]

;cursor = cnx.cursor(dictionary=True)
:Create a cursor that returns rows as [[Python:DataTypes#Dictionary_or_dict | dictionary]] { column1: value1, column2: value2 }

;cursor = cnx.cursor(buffered=True)
:Create a cursor that retrieves the complete result at once so you can check rowcount immediately. Beware of overloading your memory if you expect large results.

;cursor.execute(query)
:Execute 'query'. Traverse the cursor to fetch the data. The cursor returns a tuple for each row by default.<br>
:;cursor.rowcount
::The number of rows fetched (SELECT) or affected (INSERT,UPDATE) by this query.
:;cursor.column_names
::The columnnames

;cursor.fetchone()
:Return the next row from the cursor result.

;cursor.fetchmany(size=2)
:Return the first two rows of the result in a list.

;cursor.fetchall()
:Return all (remaining) rows of the result in a list.

<syntaxhighlight lang=python>
for row in cursor:
print(row)
for column in row:
print(column)
</syntaxhighlight>

;cursor.close()
:Close the cursor
;cnx.close()
:Close the database connection

Code example showing all:
<syntaxhighlight lang=python>
import mysql.connector

cnxconfig = {
'host': 'hostname',
'port': 3306,
'user': 'username',
'password': '',
'database': 'DBname'
}

#cnx = mysql.connector.connect(host='hostname', port=3306, user='username', database='DBname')
cnx = mysql.connector.connect(**cnxconfig)

if cnx:
print("Connection, continue")
else:
print("No connection, exit")
exit()

print("Tuple cursor")
cursor = cnx.cursor()

query = ("SELECT * FROM table")

cursor.execute(query)
for columname in cursor.column_names:
print(columname)
for row in cursor:
print(row)
for column in row:
print(column)

print("Dictionary cursor")
cursor = cnx.cursor(dictionary=True)
cursor.execute(query)
for row in cursor:
for columname in row.keys():
print("{0:25s}: {1:}".format(columname, row[columname]))

cursor.close()
cnx.close()
</syntaxhighlight>

=mariadb=
The mariadb module works almost the same as the mysql.connecctor above.

<syntaxhighlight lang=python>
import mariadb

# Connection parameters
config = {
'host': 'xxx',
'port': 3306,
'user': 'user',
'password': 'password',
'database': 'db'
}

# Connect to MariaDB
conn = mariadb.connect(**config)
cursor = conn.cursor()
cursor.execute(<query>)
for row in cursor:
print(row)
</syntaxhighlight>

The rows are tuples with a counter (id) as first element. <code>'dictionary': True</code> in the connection config should return dictionaries
;cursor.description
:Spec of the tuples in cursor
;cursor.metadata['field']
:The fields in the cursor

= ODBC connections =

<syntaxhighlight lang=python>
import pyodbc

dsn = 'DSN=<name of ODBC-DNS>'

cnx = pyodbc.connect(dsn,autocommit=True)
cursor = cnx.cursor()

query = 'show databases'

result = cursor.execute(query)
columnnames = [e[0] for e in cursor.description]
for row in result:
field1 = row[0]

</syntaxhighlight>

Works with [[Pandas#Reading Data|pandas.read_sql]] too.

= MySqlAlchemy =

For more information check the sqlalchemy.org basic usage page[https://docs.sqlalchemy.org/en/11/core/connections.html#basic-usage].

Very basic example for mysql with the pymysql module
<syntaxhighlight lang=python>
import pymysql
import sqlalchemy

dbconnect = 'mysql+pymysql://user:password@databaseserver/databasename'

engine = sqlalchemy.create_engine(dbconnect)

query = 'select something from table'
result = engine.execute(query)
for row in result:
field1 = row[0]

</syntaxhighlight>

Podman

2026-01-14T12:50:06Z

Hdridder:

[[Category:Linux/Unix]]
Podman is used to create containers, like [[Docker]]. In a container system resources are separated from each other e.g. to create a development environment where the developer can experiment freely without the risk of jeopardizing the host system.

The main difference is podman does not need a daemon like docker. It is lighter and more secure [https://www.geeksforgeeks.org/devops/podman-vs-docker/#podman-vs-docker-a-detailed-comparison]

A containerfile defines what will be in the podman image, running the image will create a container.

Podman containterfiles have the same syntax as dockerfiles. The podman build command looks first for Containerfile and then for Dockerfile in the specified directory.

=Basics=
;apt install podman
:Install podman

;This containerfile creates a working container with a python virtual environment
<syntaxhighlight lang=text>
FROM debian
LABEL name="Hans de Ridder"
RUN apt-get update
RUN apt-get -y install python3-full
RUN python3 -m venv /home/ubuntu/venv
ENV PATH="/home/ubuntu/venv/bin:$PATH"
RUN pip install pandas
CMD ["echo", "I'm alive"]
</syntaxhighlight>

;<nowiki>podman build -t <name>:<tag> <directory></nowiki>
:This builds the image with a name and a tag
;podman pull <registry image>
:Create an image from the registry image (see [[#FROM]])

;podman image ls
:List all existing images
;podman image inspect <image ID>
:Return the whole image specification as json-string

;<nowiki>podman run -v <hostdir>:<containerdir> --rm -it <image ID> bash</nowiki>
:Create the container and open a bash prompt in it (as root user). When leaving the shell the container is closed
:This way you can run any command in a container that only exists while the command is running.
:<hostdir> is mounted on <containerdir> which will be created automatically.
:NOTE: <hostdir> should not be a NFS filesystem, the container has no access to that.
;podman run -it --network=host <image ID>
:You will enter the container with the network settings (interface, hostname,...) from the host system. This enables mailing from the container.
;podman run --rm <image ID> <command>
:Run <command> in the container, remove the container on exit,
;podman exec -it <containerid> <command>
:Run <command> in a running container, -it is needed for interactive commands like bash
;podman stop <containerID>
:Stop a container
;podman rm <containerID>
:Remove a container
;podman image rm <imageID>
:Remove an image

;podman ps -a
:List all containers

;podman system prune
:Delete unused images and build caches
;podman system prune -a
:Delete unused images and all build caches

=Containerfile commands=

==FROM==
;FROM <Imagelocation>
:Specifies what is the base-image. Images are on a registry server like docker.io. You can use aliases like ubuntu or debian but that is not advised as you cannot be 100% sure where the images will be coming from. Therefor it is advised to user the full image path like docker.io/library/debian
;podman search docker.io/library/nginx
:List all images for nginx

==RUN==
;RUN <command>
:Specify a command to run when the images is build. You can have a whole lot of these.

==CMD==
;CMD <command>
:Command to run when the container starts

==LABEL==
;LABEL name="My Name"
:Give the container a label. You can have several labels of a container.

==COPY==
;COPY <localfile> <containerpath>
:Copy <localfile>, that must be in Containerfile directory, to <containerpath>

=Full-fledged container=

==Containerfile==

<syntaxhighlight lang=text>
FROM docker.io/library/debian
LABEL name="Full-fledged debian image"
LABEL maintainer="Hans de Ridder"
RUN apt-get update
RUN apt-get -y install less joe vim exim4 procps
RUN apt-get -y install python3-full
RUN python3 -m venv /opt/python/venv
ENV PATH="/opt/python/venv/bin:$PATH"
RUN pip install pandas
COPY update-exim4.conf.conf /etc/exim4/
RUN update-exim4.conf
CMD ["bash", "-i"]
</syntaxhighlight>

==Usage==
;podman build -t debian <directory>
:Build the image
;podman run -it --name FF <image ID>
:You will enter the container.
;podman start -ia FF
:Re-enter the container after you left it. If the container is already running you will be in the same shell.

Find

2026-01-13T13:10:58Z

Hdridder:

[[Category:Linux/Unix]]
Find things in a directory tree

;<code>find <directory> -type f -name '<pattern>' </code>
:Find files in and below <directory> that match <pattern>. Type f is for files, d for directory's, no type finds all.

;<code>find -L <directory>' </code>
:Do follow symbolic links, default behavior is not to follow.

;<code>find <directory> -xdev -name <pattern>' </code>
:Limit the search to the current file-system, do not go into mounted filesystems.

;<code>find <directory> -type f -name '<pattern>' -execdir <command> {} \;</code>
:For each file found execute <command> in the directory containing the file. {} represents the found file. NOTE!! Do not forget the \; at the end.
: -exec works too but is less secure as mistakes main lead to execution in an unexpected place.
: If more commands need to be executed specify -exec <command> {} \; for each command.

;<code>find <directory> -type f -mtime 1 </code>
:Find files modified 1 day ago (between 24 and 48 hours).
:Negative numbers find all files modified max 1 day ago.
:Positive numbers (explicitly signed with +) find all files at modified at least 1 day ago.
:<code>-mmin</code> counts minutes instead of days.

;<code>find <directory> -type f -newer <filename></code>
:Find files modified after the modification time of <filename>
:You can set the modification time of <filename> like <code>touch -t YYYYmmddHHMM <filename></code>

;<code>find <directory> -type f -printf '%T@ %p\n'</code>
:Print the files modification times in [https://www.epochconverter.com/ epoch] (seconds since Jan 1, 1970) and the filename. Can be used e.g. to find the last files modified (with sort -n and tail)

;<code>find <dir> -printf '%s %p\n'| sort -nr | head -10</code>
:Find the top 10 largest files under a directory

;<code>find <dir> -type f -mtime +30 -ls|awk 'BEGIN {} {totsize+=$7} END { print totsize/1000000}'</code>
:Find the total size of all files older than 30 days.

;<code>find <dir> -type f -inum <inode number> -delete</code>
:Find file by inode number and delete it. This can be used to removed files with weird names. This does not delete the inode, if files have more hardlinks all must be deleted.

Other options to select specific files are (among many others):
-user (username)
-uid (user id)
-group (group name)
-gid (group id)

;<code>find <dir> -depth -type d -empty -delete</code>
:Find empty directories and delete them

Nftables

2026-01-12T21:31:16Z

Hdridder:

[[category:networking]]
[[category:Linux/Unix]]
nftables is the successor of [[iptables]] for managing firewall rules.

At first glance [https://wiki.debian.org/nftables] it looks a bit like a [[iptables#ufw|ufw]]

{| class="wikitable" style="width:35%"
|+ Address families
! Family
! Remark
|-
| ip
| IPv4 This is the default if no family is specified
|-
| ip6
| IPv6
|-
| inet
| IPv4 + IPv6
|-
| arp
| arp
|-
| bridge
| bridge
|}

=Install and activate=
Debian comes with nftables installed since Buster.
;apt install nftables
:Install if needed
;systemctl enable nftables.service
:Create the service
;systemctl start nftables
;service nftables start
:Start the service
;systemctl status nftables
;service nftables status
:Show status

=Confiuration=
The configuration is in /etc/nftables.conf

=Commands=
;nft list table <family> filter
:Show the current filters for <family>.

Ubuntu

2026-01-07T21:48:21Z

Hdridder: /* NetworkManager */

[[Category:Linux/Unix]]
[https://www.ubuntu.com/ Ubuntu] is a linux distribution derived from [https://www.debian.org/ debian]. It's a very nice plug&play OS with all features I need.

;cat /etc/os-release
;cat /etc/lsb-release
;lsb_release -a
:Show the currently installed version. os-release has most information.

==Modfications made==

The few things I needed to tweak are listed here.

===NetworkManager===

[[NetworkManager]] is a bad idea, just use <code>/etc/network/[[Networking#Interfaces|interfaces]]</code> and /etc/resolv.conf to configure your network. Having a DHCP server to supply network configuration is fine, everything else is making things complex.

* Make sure you have ifupdown installed
* Edit /etc/network/interface. See [[Networking#Interfaces]].
* <code>sudo apt-get purge network-manager</code>
* If installed uninstall openresolv
These 2 needed to be done on a raspberry pi running debian buster
* uninstall wicd
* disable avahi (systemctl disable avahi-daemon)

===Snap===
Another bad and confusing idea is snap. E.g. [[Jupyter]] I could not get working the way I want it with the Jupyter snap.
Remove it like described by [https://techwiser.com/remove-snap-ubuntu/ techwiser].
First remove all snaps, then removed snap itself and make sure it will not reappear.
<syntaxhighlight lang=bash>
snap list
snap remove --purge <package-name>
rm -rf /var/cache/snapd/
apt autoremove --purge snapd gnome-software-plugin-snap
rm -fr ~/snap
apt-mark hold snapd
</syntaxhighlight>

When using this anyhow. Some snaps (like firefox) may send daily warnings an update is pending. For some reason this does not happen automatically and you need to execute <code>sudo snap refresh</code> to do the update.

The snap-store cannot be updated from the ubuntu software catalog as the process is running always. To update:<br>
<code>snap-store --quit && sudo snap refresh snap-store</code>

===Settings GUI===

One day I could not start the settings GUI anymore.
Removing the unity-control-center fixed the problem one time. On another install it didn't.

<code>sudo apt-get remove unity-control-center</code>

===Display===
When I connect a monitor to my laptop on HDMI automatically a 3<sup>rd</sup> VGA display is found.
;xrandr --listactivemonitors
:Show all active monitors

;xrandr --output VGA-1-2 --off
:Turn off the VGA display. The name may differ on other configurations.

===Slow boot===
# If /var/log/syslog is full of messages like '<code>failed to bind to /tmp/.X11-unix/X....</code>', make sure / is owned by root.
# Check [[Networking#Interfaces]] for allow-hotplug

===Tracker===
Tracker is a part of nautilus scanning your filesystem. I don´t need that.
apt purge nautilus
apt install thunar

===Get a visible border on the gnome terminal===
vi .config/gtk-3.0/gtk.css

terminal-window notebook {
border-width: 0px 1px 1px 1px;
border-style: solid;
border-color: grey;
}

terminal-window.maximized notebook,
terminal-window.fullscreen notebook {
border-style: none;
}

==Application management==

;apt install <package>
:Install a new package
;apt remove --purge <package>
:Remove a package and all related files
;apt update
;apt full-upgrade
:Update the package meta-data and update all
;apt-mark hold <package>
:Prevent updates of <package>, they will be show as upgradable by <code>apt update</code> but will be held back by <code>apt (full-)upgrade</code>
;apt-mark unhold <package>
:Allow updates on <package> again
;apt-mark showhold
:SHow all packages that are held back

;apt-get autoclean
:Remove all obsolete package files from /var/cache/apt/archives. Do this every now and then

;do-release-upgrade
:Upgrade to the latest Ubuntu release, latest LTS if that is what you run.

;dpkg -i <packagefile>
:Install from the packagefile

====apt update reports NO PUBKEY====
When apt update reports things like: "The following signatures couldn't be verified because the public key is not available: NO_PUBKEY [KEY_ID]" you can repair as described on linuxblog.io[https://linuxblog.io/how-to-fix-gpg-no_pubkey-error-the-following-signatures-couldnt-be/]:

<code>gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys [KEY_ID]</code>

Then store it in you keyrings directory

<code>gpg --export [KEY_ID] | sudo tee /etc/apt/keyrings/ubuntu.gpg </code>

In /apt/sources.d/xxxxx.list make sure 'Signed By' points to the .gpg file or to have '[signed-by=/etc/apt/keyrings/ubuntu.gpg]' when you have a 1-line source file

The same thing works for Debian too. I would expect Debian a keyserver of its own but I've not found it yet.

===Add applications===
To add an application to the 'Open with' list edit /usr/share/applications/<newapp>.desktop.

For opening jupyter notebooks I added jupyter.desktop that looks like below.
I don't think everything is needed but it works.

[Desktop Entry]
Type=Application
Version=1.0
Name=Jupyter
GenericName=Jupyter notebook
Comment=Prototyping and documentation
Exec=jupyter notebook %F
Icon=jupyter
Terminal=false
Categories=GTK;Development;IDE;
MimeType=text/plain;text/x-python;
StartupNotify=true
Keywords=Python;Editor;

===Autostart applications===
To automatically start an application when a user logs in put a link to /usr/share/applications/<newapp>.desktop or a new <newapp>.desktop in ~/.config/autostart

===Problem upgrading package like 'errors were encountered while processing'===
It is pretty old but some good hints can be found on [https://journalxtra.com/linux/fixing-the-dreaded-errors-were-encountered-while-processing-errors/ journalxtra.com]

Ubuntu

2026-01-07T19:43:13Z

Hdridder:

[[Category:Linux/Unix]]
[https://www.ubuntu.com/ Ubuntu] is a linux distribution derived from [https://www.debian.org/ debian]. It's a very nice plug&play OS with all features I need.

;cat /etc/os-release
;cat /etc/lsb-release
;lsb_release -a
:Show the currently installed version. os-release has most information.

==Modfications made==

The few things I needed to tweak are listed here.

===NetworkManager===

[[NetworkManager]] is a bad idea, just use <code>/etc/network/[[Networking#Interfaces|interfaces]]</code> and /etc/resolv.conf to configure your network. Having a DHCP server to supply network configuration is fine, everything else is making things complex. You may need to install ifupdown and disable the netplan thing to achieve this.
* sudo apt-get purge network-manager
* Uninstall dhcpcd
* Uninstall openresolv
These 2 needed to be done on a raspberry pi running debian buster
* uninstall wicd
* disable avahi (systemctl disable avahi-daemon)

===Snap===
Another bad and confusing idea is snap. E.g. [[Jupyter]] I could not get working the way I want it with the Jupyter snap.
Remove it like described by [https://techwiser.com/remove-snap-ubuntu/ techwiser].
First remove all snaps, then removed snap itself and make sure it will not reappear.
<syntaxhighlight lang=bash>
snap list
snap remove --purge <package-name>
rm -rf /var/cache/snapd/
apt autoremove --purge snapd gnome-software-plugin-snap
rm -fr ~/snap
apt-mark hold snapd
</syntaxhighlight>

When using this anyhow. Some snaps (like firefox) may send daily warnings an update is pending. For some reason this does not happen automatically and you need to execute <code>sudo snap refresh</code> to do the update.

The snap-store cannot be updated from the ubuntu software catalog as the process is running always. To update:<br>
<code>snap-store --quit && sudo snap refresh snap-store</code>

===Settings GUI===

One day I could not start the settings GUI anymore.
Removing the unity-control-center fixed the problem one time. On another install it didn't.

<code>sudo apt-get remove unity-control-center</code>

===Display===
When I connect a monitor to my laptop on HDMI automatically a 3<sup>rd</sup> VGA display is found.
;xrandr --listactivemonitors
:Show all active monitors

;xrandr --output VGA-1-2 --off
:Turn off the VGA display. The name may differ on other configurations.

===Slow boot===
# If /var/log/syslog is full of messages like '<code>failed to bind to /tmp/.X11-unix/X....</code>', make sure / is owned by root.
# Check [[Networking#Interfaces]] for allow-hotplug

===Tracker===
Tracker is a part of nautilus scanning your filesystem. I don´t need that.
apt purge nautilus
apt install thunar

===Get a visible border on the gnome terminal===
vi .config/gtk-3.0/gtk.css

terminal-window notebook {
border-width: 0px 1px 1px 1px;
border-style: solid;
border-color: grey;
}

terminal-window.maximized notebook,
terminal-window.fullscreen notebook {
border-style: none;
}

==Application management==

;apt install <package>
:Install a new package
;apt remove --purge <package>
:Remove a package and all related files
;apt update
;apt full-upgrade
:Update the package meta-data and update all
;apt-mark hold <package>
:Prevent updates of <package>, they will be show as upgradable by <code>apt update</code> but will be held back by <code>apt (full-)upgrade</code>
;apt-mark unhold <package>
:Allow updates on <package> again
;apt-mark showhold
:SHow all packages that are held back

;apt-get autoclean
:Remove all obsolete package files from /var/cache/apt/archives. Do this every now and then

;do-release-upgrade
:Upgrade to the latest Ubuntu release, latest LTS if that is what you run.

;dpkg -i <packagefile>
:Install from the packagefile

====apt update reports NO PUBKEY====
When apt update reports things like: "The following signatures couldn't be verified because the public key is not available: NO_PUBKEY [KEY_ID]" you can repair as described on linuxblog.io[https://linuxblog.io/how-to-fix-gpg-no_pubkey-error-the-following-signatures-couldnt-be/]:

<code>gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys [KEY_ID]</code>

Then store it in you keyrings directory

<code>gpg --export [KEY_ID] | sudo tee /etc/apt/keyrings/ubuntu.gpg </code>

In /apt/sources.d/xxxxx.list make sure 'Signed By' points to the .gpg file or to have '[signed-by=/etc/apt/keyrings/ubuntu.gpg]' when you have a 1-line source file

The same thing works for Debian too. I would expect Debian a keyserver of its own but I've not found it yet.

===Add applications===
To add an application to the 'Open with' list edit /usr/share/applications/<newapp>.desktop.

For opening jupyter notebooks I added jupyter.desktop that looks like below.
I don't think everything is needed but it works.

[Desktop Entry]
Type=Application
Version=1.0
Name=Jupyter
GenericName=Jupyter notebook
Comment=Prototyping and documentation
Exec=jupyter notebook %F
Icon=jupyter
Terminal=false
Categories=GTK;Development;IDE;
MimeType=text/plain;text/x-python;
StartupNotify=true
Keywords=Python;Editor;

===Autostart applications===
To automatically start an application when a user logs in put a link to /usr/share/applications/<newapp>.desktop or a new <newapp>.desktop in ~/.config/autostart

===Problem upgrading package like 'errors were encountered while processing'===
It is pretty old but some good hints can be found on [https://journalxtra.com/linux/fixing-the-dreaded-errors-were-encountered-while-processing-errors/ journalxtra.com]

NetworkManager

2026-01-07T19:40:49Z

Hdridder:

[[Category:Linux/Unix]]
[[Category:Networking]]
I prefer using old style network configuration as described in [[Networking#Interfaces]]. How to disable it is described in [[Ubuntu#NetworkManager]].

Some distributions come with NetworkManager that provides a working standard configuration. If you want to use it, here some tricks to do non-standard things with it.

Configuration is stored in /etc/NetworkManager/system-connections/ in a file per connection.

To specify addresses use below. In the ipv6 section put auto to have the full SLAAC configurationon the interfaces to. With manual only the addresses specified and the routes provided my Route Advertisement are configured.
<syntaxhighlight lang=text>
[ipv4]
address1=<CIDR>, <gateway>
dns=<DNSserverIP>;
dns-search=<search domain>;
method=manual

[ipv6]
addr-gen-mode=default
address1=<CIDR>,<gateway>
dns=<DNSserverIP>;
dns-search=<serach domain>;
method=auto
</syntaxhighlight>

IPv4

2026-01-04T21:15:37Z

Hdridder: Created page with "Category:networking Category:Linux/Unix IPv4 is the original addressing scheme for IP networks. IPv4 addresses are 32 bits long so that is some more than 4 billion addresses. That is a lot but not enough when everything that needs communication get at least 1 address. That is why IPv6 has been invented that has 128 bit addresses that is really a lot (3.402823669×10³⁸) An address looks like aaa.bbb.ccc.xxx The netmask defines how many addresses are wit..."

[[Category:networking]]
[[Category:Linux/Unix]]
IPv4 is the original addressing scheme for IP networks.

IPv4 addresses are 32 bits long so that is some more than 4 billion addresses. That is a lot but not enough when everything that needs communication get at least 1 address. That is why [[IPv6]] has been invented that has 128 bit addresses that is really a lot (3.402823669×10³⁸)

An address looks like aaa.bbb.ccc.xxx

The netmask defines how many addresses are within the subnet and where packets are routed to. Addresses within the subnet are routed to the interface, address outside the subnet to the default gateway (router).
Net netmask is traditionally expressed in the same format as an address (e.g. 255.255.255.0), later CIDR was invented that just specified the number of bits that specify the subnetwork.

{| class="wikitable"
! Class
! Addresses
! Netmask
! CIDR
! Remark
|-
| A
| style="text-align:right"| 1-126.xxx.xxx.xxx
| 255.0.0.0
| /8
|
|-
| A private
| style="text-align:right"| 10.xxx.xxx.xxx
| 255.0.0.0
| /8
| Private range, not routed
|-
| B
| style="text-align:right"| 128-191.xxx.xxx.xxx
| 255.255.0.0
| /16
|
|-
| B private
| style="text-align:right"| 172.16-31.xxx.xxx
| 255.255.0.0
| /16
| Private, not routed
|-
| C
| 192-223.xxx.xxx.xxx
| 255.255.255.0
| /24
|
|-
| C private
| style="text-align:right"| 192.168-xxx.xxx
| 255.255.255.0
| /24
| Private, not routed
|-
| D
| style="text-align:right"| 224-239.xxx.xxx.xxx
|
|
| Multicasting addresses, not for host assignment
|-
| E
| style="text-align:right"| 240-255.xxx.xxx.xxx
|
|
| Reserved addresses, not for public use
|}

IPv6

2026-01-04T20:32:03Z

Hdridder:

[[Category:networking]]
[[Category:Linux/Unix]]
Testing your IPv6 connection:
;https://ifconfig.co/
:Including CLI-support
;https://ready.chair6.net/
:Nice as simple interface
;https://test-ipv6.com/
Testing website security and connectivity
;https://internet.nl

Below seems to be broken December 2025
;https://ipv6-test.com/
:Check your IPv6 configuration
;http://ipv6-test.com/validate.php
:Check IPv6 support for websites

==Address Assignment==

The [http://www.iana.org/assignments/ipv6-address-space IPv6 address assignment] is handled by IANA. Most important prefixes are:
<table class="plain">

<tr>
<td>2000::/3</td>
<td>Global Unicast</td>
<td>Global scope</td>
</tr>
<tr>
<td>FC00::/7</td>
<td>Unique Local Unicast</td>
<td>Local Unicast. For intranets, not routed, like IPv4 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Currently FC00::/8 is undefined, only FD00::/8 is used</td>
</tr>
<tr>
<td>FE80::/10</td>
<td>Link Local Unicast</td>
<td>Link Identification, addresses can be based on the interface MAC address</td>
</tr>
<tr>
<td>FF00::/8</td>
<td>Multicast</td>
<td></td>
</tr>

</table>

==Local Unicast (ULA)==
Users were advised to generate a 40 bits Global ID for their intranets but it is unclear if there a valid use case for it[[https://en.wikipedia.org/wiki/Unique_local_address]]. It is advised to calculate the Global Id using a specified procedure to maximize the probability of an unique address.

<strong>[http://tools.ietf.org/html/rfc4193#section-3.2 Local Unicast Address-format]:</strong>
<table>
<tr>
<th align=left>Prefix</td>
<th align=left>L</th>
<th align=left>Global ID</th>
<th align=left>Subnet ID</th>
<th align=left>Interface ID</th>
</tr>
<tr>
<td>7 bits<br />
FC00::/7</td>
<td>1 bit<br />
Locally assigned</td>
<td>40 bits<br />
User defined</td>
<td>16 bits<br />
Local subnet</td>
<td>64 bits<br />
Locally assigned or by dhcp</td>
</tr>
<tr>
<td>1111110</td>
<td>1</td>
<td></td>
<td></td>
<td></td>
</tr>

</table>
The Interface ID is in the so-called ‘Modified EUI-64 Format’. How to go from a MAC address to Modified EUI-64 format is described in appendix A of [href="http://tools.ietf.org/html/rfc4291 RFC 4291]. When using ULA prefixes you must have Routing Advertising (radvd) to tell the network it should configure addresses and routes for it

==Route Advertisement==
Provides information for ‘Stateless Autoconfiguration’ to the local nodes.

IPv6 systems can send a Route Solicitation (RS) messages. These are answered by a Route Advertisement (RA) message that tells with prefixes exist and, when needed where the gateway for that prefix is. The client then can configure IPv6 addresses and routes.

On linux systems there is the radvd deamon to provide RA.

To advertise there is a ULA prefix in the network put this in /etc/radvd.conf
<syntaxhighlight lang=text>
interface eth0
{
AdvSendAdvert on;
AdvDefaultLifetime 0;
prefix fdxx:xx:xx::/48
{
};
};
</syntaxhighlight>

==Disable IPv6 on a system==

At the end of <code>/etc/sysctl.conf</code> add:

<syntaxhighlight lang=bash>
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
</syntaxhighlight>

;sysctl -p
:Force reload of the kernel parameters as in /etc/sysctl.conf

;cat /proc/sys/net/ipv6/conf/all/disable_ipv6
:Check if IPv6 has been disabled:
:0 => IPv6 is not disabled
:1 => IPv6 is disabled

==Check IPv6 autoconfiguration==
;<code>cat /proc/sys/net/ipv6/conf/eth0/autoconf</code>
:0 => autoconf is disabled
:1 => autoconf is enabled

==Misc commands==

;/sbin/ip -6 addr del <ipv6address>/<prefixlength> dev <interface>
:Remove ipv6 addres from an interface

Dhcp

2026-01-04T20:31:35Z

Hdridder:

[[Category:networking]]
[[Category:Linux/Unix]]
Dynamic Host Configuration Protocol

Provide IP addresses for your network from a central server.
==IPv6==

[[IPv6]] addresses can be advertised by the radvd (often on your ISP's router). These are based on the MAC address of the client or pseudo randomly generated. For more control over the addresses you can run a dhcp server. When the [https://www.isc.org/dhcp/ ISC dhcp server] is used you need separate instances for the IPv4 and the IPv6 server.

=KEA=
KEA is the successor of the isc-dhcp-server.
;apt install kea
:Install all packages for the KEA-server

=isc-dhcp-server=
Sample configuration file (/etc/dhcp/dhcpd6.conf).
<syntaxhighlight lang=bash>
log-facility local7;
default-lease-time 86400;
max-lease-time 604800;

subnet6 your:site:ipv6:prefix::/64 {
# range6 your:ipv6:prefix:address:1::f000 your:ipv6:prefix:address:1::ffff;

option dhcp6.domain-search "local.domain.tld";
option dhcp6.name-servers <name-server-address>;
option ntp-servers ntp.domain.tld;

host host1name {
host-identifier option dhcp6.client-id 00:a4:00:01:45:26:4d:8e:98:8e:76:d1:09:32;
fixed-address6 <full ipv6-address>;
}
</syntaxhighlight>
Hosts for fixed addresses are found by their DUID. Only way I know to find it on the server is from the logfile (/var/log/syslog). On the client it is in var/lib/dhcpcd/duid

The firewall on the client needs to be open on the Link Local address (in the FE80::/10) range on port 546

==IPv4==
Sample configuration file (/etc/dhcp/dhcpd.conf).
<syntaxhighlight lang=bash>

#
ddns-update-style none;

# option definitions common to all supported networks.
option domain-name "domain.tld";
option domain-name-servers 192.168.1.2, 208.67.222.222;
option routers 192.168.1.254;
option broadcast-address 192.168.1.255;
option ntp-servers 192.168.1.4;

# 1 day (fine for home networks, e.g. hotspots should have far shorter lease times)
default-lease-time 86400;
# 1 week
max-lease-time 604800;

authoritative;

log-facility local7;

# Known hosts get fixed IP address defined in DNS-server
group {
deny unknown-clients;
# One week, these are sort of fixed addresses.
default-lease-time 604800;

host host1name {
hardware ethernet ff:ff:ff:ff:ff:ff;
fixed-address host1name.domain.tld;
}
host host2name {
hardware ethernet ff:ff:ff:ff:ff:ff;
fixed-address host2name.domain.tld;
}
}
# All other get an address from the range
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.100 192.168.1.199;
}
</syntaxhighlight>

The leases are kept track of in /var/lib/dhcp/dhcpd.leases
An ordinary entry looks like:
tstp and cltt are for the failover protocol, most other entries are self-explaining I think.
<syntaxhighlight lang=bash>
lease 192.168.1.133 {
starts 6 2021/03/18 11:22:00;
ends 0 2021/03/19 11:22:00;
tstp 0 2021/03/19 11:22:00;
cltt 6 2021/03/19 11:22:00;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet ff:ff:ff:ff:ff:ff;
client-hostname "host3name";
}
</syntaxhighlight>

=DHCP clients=
;dhcpcd -k <interface>
:Release lease
;dhcpcd -n <interface>
:Renew lease

=Traffic check=
;tcpdump -i eth0 port 547
:Check on server side
;tcpdump -i eth0 port 546
:Check on client side

Networking

2026-01-04T20:17:29Z

Hdridder:

[[Category:networking]]
[[Category:Linux/Unix]]

=Basics=
For IP networks there are 2 address schemes, [[IPv4]] and [[IPv6]]
;ping <address>
:Get response from <address> interface
;<code>for i in $(seq 1 254); do ping -c1 -t 1 10.117.192.$i; done</code>
:Scan a subnet

=Check network connectivity=

;netstat -na # deprecated
;ss -na
:Show port status (LISTEN, ESTABLISHED, .....)
:If you are root -p adds the process that owns the socket

;<code>nc -v -z <host> <port></code>
:Netcat: Tell me if we can open a connection to <host> on <port>. Without -z it works like telnet

;<code>nc -v -u -z <host> <port></code>
:Tell me if we can open a connection to <host> on a UDP <port>

;<code>curl -vvv -k --proxy1.0 <proxyIP>:<proxyPort> <Remote URL></code>
:Get the data from <remote URL> via <proxyIP> -k allows insecure SSL connections

;<code>curl -vvv telnet://<IP>:<port></code>
:Just test connectivity to a port

<syntaxhighlight lang=bash>
#!/bin/bash

PORT=$1

TARGETS="999.999.999.999
888.888.888.888
777.777.777.777
666.666.666.666
"

for TARGET in ${TARGETS}
do
curl http://${TARGET}:${PORT} --connect-timeout 1 > /dev/null 2>&1
if [ $? == 28 ] # 28 is the curl code for connection timeout
then
echo No connection to ${TARGET}:${PORT}
fi
done
</syntaxhighlight>

=Interfaces=
;ifconfig -a # Deprecated, replaced by ip
:Show all network interfaces

;ifconfig <interface> <ipaddress>
:Assign <ipaddress> to <interface>

;ip address
:Show all IP addresses
;ip -6 address
:Show all IPv6 addresses

;ip -6 addr add <ipv6address>/<prefixlength> dev <interface>
:Add an IPv6 address to <interface>

;mii-tool <interface>
:Show interface link status (speed)

When not using confusing things like [[NetworkManager]] network configuration is controlled by the interfaces file.
;/etc/network/interfaces
<syntaxhighlight lang=text>
auto <interface> # Boot process waits until the interface is up or a time-out (5 minutes)
allow-hotplug <interface> # Boot process does not wait for the interface.
iface <interface> inet dhcp
wpa-ssid <ssid>
wpa-psk <password>

# For static ipv4
interface <interface> inet static
address xxx.xxx.xxx.xxx/<masklen>
gateway xxx.xxx.xxx.xxx
# To have ipv6 auto-configured (by SLAAC)
iface <interface> inet6 auto

# To have ipv6 auto-configured (by SLAAC) and add own address
iface <interface> inet6 static
address unique:local:address:prefix:your:own:host:address
autoconf 1
netmask 64

# Add another address to <interface>(work for IPv4 addresses too
iface <interface> inet6 static
address unique:local:address:prefix:your:own:host:address
netmask 64

# To have ipv6 auto-configured (by SLAAC) and query a dhcp-server
iface <interface> inet6 dhcp
autoconf 1
</syntaxhighlight>

;Disable an interface completely (even not shown by ifconfig)
:vi /etc/modprobe.d/blackllst.conf
:Put in the driver for your interface like: <code>blacklist ath9k</code>

=Wireless=
;iwscan <interface> scanning
:Show available accesspoints
;iwconfig <interface>
:Show details of the wireless connection

=Routing=
;route add -net default gw <IPaddress> # Deprecated
;ip route add -net default gw <IPaddress>
:Add a default route using <IPaddress> as gateway

=Tunneling=
;<code>ssh -L 8080:remotehost:3006 proxy</code>
:Login over ssh to proxy and there connect to remotehost port 3006. Connect local port 8080 to the remote connection.
:Now if you connect to port 8080 on your local server you are connected to port 3006 on the remotehost
:Using <code>-nNT</code> prevents that the ssh command opens a terminal on the proxy. The ssh command however must remain active.

=Tracing=
For tracing traffic see [[Tcpdump]]

;traceroute <destination>
:Show which route a package takes to reach destination

;ip route get <IPaddress>
:Show the interface used to go to <IPaddress>
To get just the interface name you can use this[https://unix.stackexchange.com/questions/14961/how-to-find-out-which-interface-am-i-using-for-connecting-to-the-internet]:
<syntaxhighlight lang=bash>
host=<fqdn>
host_ip=$(getent ahosts "$host" | awk '{print $1; exit}')
ip route get "$host_ip" | grep -Po '(?<=(dev ))(\S+)'
</syntaxhighlight>

=Other=
;<code>wget <URL></code>
:Get the data from <URL>

Networking

2026-01-04T20:12:10Z

Hdridder: /* Interfaces */

[[Category:networking]]
[[Category:Linux/Unix]]

=Basics=
;ping <address>
:Get response from <address> interface
;<code>for i in $(seq 1 254); do ping -c1 -t 1 10.117.192.$i; done</code>
:Scan a subnet

=Check network connectivity=

;netstat -na # deprecated
;ss -na
:Show port status (LISTEN, ESTABLISHED, .....)
:If you are root -p adds the process that owns the socket

;<code>nc -v -z <host> <port></code>
:Netcat: Tell me if we can open a connection to <host> on <port>. Without -z it works like telnet

;<code>nc -v -u -z <host> <port></code>
:Tell me if we can open a connection to <host> on a UDP <port>

;<code>curl -vvv -k --proxy1.0 <proxyIP>:<proxyPort> <Remote URL></code>
:Get the data from <remote URL> via <proxyIP> -k allows insecure SSL connections

;<code>curl -vvv telnet://<IP>:<port></code>
:Just test connectivity to a port

<syntaxhighlight lang=bash>
#!/bin/bash

PORT=$1

TARGETS="999.999.999.999
888.888.888.888
777.777.777.777
666.666.666.666
"

for TARGET in ${TARGETS}
do
curl http://${TARGET}:${PORT} --connect-timeout 1 > /dev/null 2>&1
if [ $? == 28 ] # 28 is the curl code for connection timeout
then
echo No connection to ${TARGET}:${PORT}
fi
done
</syntaxhighlight>

=Interfaces=
;ifconfig -a # Deprecated, replaced by ip
:Show all network interfaces

;ifconfig <interface> <ipaddress>
:Assign <ipaddress> to <interface>

;ip address
:Show all IP addresses
;ip -6 address
:Show all IPv6 addresses

;ip -6 addr add <ipv6address>/<prefixlength> dev <interface>
:Add an IPv6 address to <interface>

;mii-tool <interface>
:Show interface link status (speed)

When not using confusing things like [[NetworkManager]] network configuration is controlled by the interfaces file.
;/etc/network/interfaces
<syntaxhighlight lang=text>
auto <interface> # Boot process waits until the interface is up or a time-out (5 minutes)
allow-hotplug <interface> # Boot process does not wait for the interface.
iface <interface> inet dhcp
wpa-ssid <ssid>
wpa-psk <password>

# For static ipv4
interface <interface> inet static
address xxx.xxx.xxx.xxx/<masklen>
gateway xxx.xxx.xxx.xxx
# To have ipv6 auto-configured (by SLAAC)
iface <interface> inet6 auto

# To have ipv6 auto-configured (by SLAAC) and add own address
iface <interface> inet6 static
address unique:local:address:prefix:your:own:host:address
autoconf 1
netmask 64

# Add another address to <interface>(work for IPv4 addresses too
iface <interface> inet6 static
address unique:local:address:prefix:your:own:host:address
netmask 64

# To have ipv6 auto-configured (by SLAAC) and query a dhcp-server
iface <interface> inet6 dhcp
autoconf 1
</syntaxhighlight>

;Disable an interface completely (even not shown by ifconfig)
:vi /etc/modprobe.d/blackllst.conf
:Put in the driver for your interface like: <code>blacklist ath9k</code>

=Wireless=
;iwscan <interface> scanning
:Show available accesspoints
;iwconfig <interface>
:Show details of the wireless connection

=Routing=
;route add -net default gw <IPaddress> # Deprecated
;ip route add -net default gw <IPaddress>
:Add a default route using <IPaddress> as gateway

=Tunneling=
;<code>ssh -L 8080:remotehost:3006 proxy</code>
:Login over ssh to proxy and there connect to remotehost port 3006. Connect local port 8080 to the remote connection.
:Now if you connect to port 8080 on your local server you are connected to port 3006 on the remotehost
:Using <code>-nNT</code> prevents that the ssh command opens a terminal on the proxy. The ssh command however must remain active.

=Tracing=
For tracing traffic see [[Tcpdump]]

;traceroute <destination>
:Show which route a package takes to reach destination

;ip route get <IPaddress>
:Show the interface used to go to <IPaddress>
To get just the interface name you can use this[https://unix.stackexchange.com/questions/14961/how-to-find-out-which-interface-am-i-using-for-connecting-to-the-internet]:
<syntaxhighlight lang=bash>
host=<fqdn>
host_ip=$(getent ahosts "$host" | awk '{print $1; exit}')
ip route get "$host_ip" | grep -Po '(?<=(dev ))(\S+)'
</syntaxhighlight>

=Other=
;<code>wget <URL></code>
:Get the data from <URL>

Raspberry Pi

2026-01-04T12:58:23Z

Hdridder: /* System configuration */

A [[https://www.raspberrypi.com/ Raspberry Pi]] is a amazingly cheap but working computer at the size of a deck of cards.

=Installation=
Put the installation imaged on a USB device stick and boot from that.<br>
After installing it works fine as as headless server but if you want to use it as your desktop system there as some thinks to take care of.<br>
You may need a lot more storage locally (e.g. to sync with your [[nextcloud|cloud-storage]]). You can plug in any USB device and use that.<br>

=System configuration=
;raspi-config
:Configuration utility

;<code>/etc/NetworkManager/system-connections</code>
:Has files with the network configuration

==Disable wifi and bluetooth==
Edit /boot/firmware/config.txt <br/>
Add following lines after the line <code># Additional overlays and parameters are documented</code>
dtoverlay=disable-wifi
dtoverlay=disable-bt

=X11 applications=
RaspiOS is using [https://wiki.debian.org/Wayland wayland] as backend for the displan server backend. Although it is said wayland is the future and does have xwayland that should allow X11 applications to run some don't. Applications will just show you 'cannot open display' or something similar

With this help [https://geeks3d.com/20240509/how-to-switch-from-wayland-to-x11-on-raspberry-pi-os-bookworm/] I found the solution.

;echo $XDG_SESSION_TYPE
:If this shows 'wayland' that is your backend

In raspi-config 'Advanced Options' - 'A6 Wayland' select X11 and reboot your system.

Cron

2026-01-01T13:41:20Z

Hdridder:

Timed execution of commands. The cron daemon is checking the crontab every minute to see if anything needs to be done

;Format of crontab

{| class="wikitable"
! minute
! hour
! day-of-month
! month
! day-of-week
! command
|-
| 1-59
| 1-23
| 1-31
| 1-12
| 0-7
| doit
|}

In scheduling fields (all except the command) following applies:

* * means each tick is matched.
* */5 means each 5th tick is matched (0,5,10,...)
* A range is defined by a dash. E.g. 2-5 in minute -> execute command on minute 2,3,4 and 5
* A list is defined by comma's E.g. 2,4,6 in minute -> execute command on minute 2, 4 and 6
* For month and day-of-week also names can be used by giving the first 3 characters (case insensitive)
* In day-of-week numbers 0 and 7 are Sunday

Example:
Every Sunday at noon execute my script:
<syntaxhighlight lang=bash>
00 12 * * sun /usr/local/bin/myscript.sh
00 12 * * 0 /usr/local/bin/myscript2.sh
</syntaxhighlight>

Who can use cron can be limited in /etc/cron.[allow|deny]
* if cron.allow exists - only users listed into it can use crontab
* If cron.allow does not exist - all users except the users listed into cron.deny can use crontab
* If neither of the file exists - only the root can use crontab
* If a user is listed in both cron.allow and cron.deny - that user can use crontab.

;crontab -l
:List the current crontab

;crontab <crontabfile>
:Load the file into the crontab, all existing content will be replaced. A user's crontab is stored in /var/spool/cron/crontabs/<username>. Don't edit it there, the cron daemon will only read it if told so by the crontab command.

;crontab -e
:Edit the current crontab directly in the default editor. Change the default editor by setting the EDITOR environment variable.