AUTOLOCATE
Posted: 18 September, 2020 Filed under: Storage schema management | Tags: autolocate, page size, row size, rows per page 3 CommentsIntroduction
AUTOLOCATE is a very interesting and relatively new Informix feature and one that I believe has been so far undersold. Introduced in 12.10.xC3, the Informix manual has this to say about it:
Use the AUTOLOCATE configuration parameter to control whether the database server controls the location of new databases, indexes, and tables and the fragmentation of those tables. If you set the AUTOLOCATE configuration parameter to a positive integer, the database server performs the following tasks:
- Stores new databases for which you do not specify a location in the optimal dbspace instead of in the root dbspace. By default, all dbspaces except dbspaces that are dedicated to tenant databases are available. However, you can control the list of available dbspaces.
- Fragments new tables by round-robin, where the number of fragments is equal to the value of the AUTOLOCATE configuration parameter.
- Adds more table fragments as the table grows.
I had ignored it, largely because on first reading it appears to be a feature aimed at newbies to Informix. Our systems are mostly well-established and decisions regarding dbspaces and fragmentation have already been made by DBAs sweating over a keyboard driving a spreadsheet (or script).
In fact it has a few highly significant and undocumented facets that address a number of fundamental storage issues. In a nutshell if you wanted a table to have the potential to grow as large as possible you needed to be aware of the following (taken from IBM support document, What is the maximum number of pages for a table or fragment? What is the maximum number of rows per page?):
- 3 bytes are used to store the page number a maximum of 16,777,215 pages can be used in one table.
- 1 byte is used to store the slot number, a single page can have at most 255 slots/rows.
- The maximum number of rows in a table or fragment is 4,278,189,825.
The last limitation is the combination of the first two at their limits and doesn’t need separate consideration, however all of this combined means:
- any table larger than 32, 64, 96, 128, 160, 196 or 256 GB, depending on the page size (2-16 kB), needs to be partitioned.
- tables with narrow rows cannot benefit from the larger page sizes without wasting storage space.
- when defining an interval for a table partitioned by range, the interval must be not so large that it allows fragments to reach the page limit.
- tables partitioned by round-robin are limited to the above size limits multiplied by the number of fragments you created.
The fundamental limits from the support document are still there and probably will be for some time to come. However, AUTOLOCATE makes it much much easier to deal with the most significant of them, as we will see.
AUTOLOCATE in action
Firstly let’s imagine a table using a 2 kB page size, fragmented by round robin with two partitions. Its maximum size is 64 GB. The term round-robin refers to how data gets inserted into the table. It will be in the fashion 1-2-1-2-1-2 etc. If the table had four partitions it would be 1-2-3-4-1-2-3-4-1-2-3-4. If your table starts from empty and you insert data over a period of time, all the partitions will contain the same number of rows. What happens if my table with two partitions approaches 64 GB, let’s say 60 GB, and I cannot delete any data and must keep inserting more? Well I can add another partition or two. (Even another ten.) My table will now will fill up 1-2-3-4-1-2-3-4 etc. However two partitions have 60 GB and two are empty so I am still 4 GB away from two of the partitions filling; it is just going to take twice as long to get there.
Let’s see how AUTOLOCATE can help by creating a table for testing and filling it with data.
Create table and populate:
drop table if exists rr_table;
create table rr_table (
myid bigserial not null,
mydata char(40) not null
) with rowids fragment by round robin in 2kb_dbspace1, 2kb_dbspace2;
create table source_data (
text_field varchar(40)
);
insert into source_data values ('r4dR3leer4IWTLNo5rZpYS9HTGrQngjcDp5FcQwn');
insert into source_data values ('FtIgPPVDPvnqlSf6QNSkkIDv7GvXIrN2L22w4x7K');
insert into source_data values ('N5VtAmIxSl89lt3O34hshvCktWjYOiMdGJjwwjVe');
insert into source_data values ('Kb13xQcI5IZqFjC0bu4u4zqK1JUSvZKjmGD1OZ3K');
insert into source_data values ('5XhqJQ1lm2LYKzatCngXrEJMEDGF8Qw5gxsetqTx');
insert into source_data values ('C1EXpTrmk28F2ETnvoK1QhF8v3gA4iBciaQw7NtW');
insert into source_data values ('7OvpiycZRjTFPEzk6FM2hJsORWVEAUhOM2O211tr');
drop procedure if exists insert_data;
create procedure insert_data ()
loop
begin work;
insert into rr_table (mydata) select * from source_data;
commit;
end loop;
end procedure;
execute procedure insert_data();
While this was running to fill up my table, I thought I would experiment with a different number of threads and values for AUTOLOCATE. Each result below is the average of five measurements taken over ten seconds. AUTOLOCATE can be changed dynamically and immediately affects running sessions. For the tests I created multiple source tables to rule out contention there.
Number of sessions inserting into two-way round-robin table, no indices | Rows inserted per second AUTOLOCATE 0 |
Rows inserted per second AUTOLOCATE 1 |
---|---|---|
1 | 85 300 | 90 300 |
2 | 94 400 | 93 400 |
3 | 110 200 | 99 700 |
4 | 117 200 | 100 500 |
5 | 121 400 | 96 000 |
6 | 120 100 | 91 300 |
7 | 115 800 | 88 800 |
8 | 109 000 | 87 000 |
(For values of AUTOLOCATE greater than 1 the results were very similar to 1 so I haven’t bothered to include any in the table.)
So we can see that AUTOLOCATE does have an effect on how data are inserted: it changes the algorithm for insertion into tables fragmented by round-robin. If the parameter is set to any non-zero value the partition which is most empty will receive the rows. This means it isn’t really round-robin any more. However, when populating a table from empty the behaviour appears to be more or less the same, however in my example once my two new partitions are added they will receive all the new rows.
What happens in the situation where one or more fragments reaches the page limit? Without AUTOLOCATE set the database engine will continue to insert data into fragments with space, however maximum insert performance will reduce by a factor of about two hundred. This is presumably because it checks bitmap pages in the full fragments each time their turn comes in case any data were deleted by other processes.
However, with AUTOLOCATE set, with the new algorithm (most empty fragment gets priority) this situation does not arise unless all fragments are full.
Furthermore with AUTOLOCATE, when all fragments are full the database engine will automatically add more. One thing to be aware of is if, say, you had a R-R table with eight fragments and they all fill, if AUTOLOCATE is set to just 1, the engine will add just one fragment so you are no longer writing into multiple fragments. However the Informix manual section Managing automatic location and fragmentation explains this and how to control the dbspace used very well.
With AUTOLOCATE set to 0, I eventually hit the dreaded 271: Could not insert new row into the table.
The online log reports it too:
WARNING: partition 'mydbname:informix.rr_table': no more pages
Adding two more fragments with ALTER FRAGMENT ADD I can now test the situation where two fragments are full and two are empty and show the basis for the two-hundred times speed reduction claim above. I suspect this will be highly dependent on the number of bitmap pages to scan and how these pages are marked. In this test, if AUTOLOCATE is 0 you’ll continue to periodically see the no more pages message in the online log.
Number of sessions inserting into four-way round-robin table with two full fragments, no indices | Rows inserted per second AUTOLOCATE 0 |
Rows inserted per second AUTOLOCATE 1 |
---|---|---|
1 | 390 | 74 220 |
2 | 588 | 83 300 |
3 | 741 | 91 900 |
4 | 919 | 96 100 |
There is a bit of a drop in performance over the initial results with a two-way R-R table, even when AUTOLOCATE is a non-zero value. An assumption you could make is that this is because there are more fragments to check the size of before inserting but I have not tested or verified this.
So you can see this is extremely useful if your round-robin fragmented tables are getting large: just add more fragments and ensure AUTOLOCATE is on; or even if you’re brave let the engine do this for you. I say brave because you should be aware that our testing found defect IT34090 which is where this process fails for tables with rowids and varchar or lvarchar columns. I hope it will be fixed in 12.10.xC15 and 14.10.xC5.
Bored of calculating the optimum page sizes for your tables?
I also mentioned that narrow tables cannot benefit from large page sizes because 255 times the row size can often come to under the page size. This means a careful DBA needs to calculate the optimum page size for any given table. There will usually be a range of page sizes that will be suitable and two approaches I have used are to take the largest that will not waste space or, if you perhaps intend to use table compression, assume your rows are two to three times less wide than they are and apply the same logic (since compression increases the number of rows per page possible but does not change the 255 rows/page limit).
However you can now let AUTOLOCATE do this job for you. The Informix manual states: stores new databases for which you do not specify a location in the optimal dbspace instead of in the root dbspace. “Optimal” is not defined but I hope it would take into account all the considerations above.
It’s quite easy to start experimenting by providing the database engine with a list of dbspaces of different page sizes and see where your data gets placed. (You will need to insert at least one row for an extent to actually be created.)
EXECUTE FUNCTION task("autolocate database", "my_dbname", "2kb_dbspace1,4kb_dbspace1,6kb_dbspace1,8kb_dbspace1,10kb_dbspace1,12kb_dbspace1,14kb_dbspace1,16kb_dbspace1");
Selected results are in the tables below. My initial expectation was that there would be clear stepping points based on the row size where the algorithm would select the next page size up and, while this is definitely an overall trend, the inner workings of the algorithm were difficult to ascertain when I only examined ten or so tables. To try and see the patterns more clearly I created around 1500 tables with different schemas and examined the page size selected versus the row size. For the test all possible page sizes were made available but your system may restrict what can be used.
The first result I found is that the page size selected seems to entirely depend on the row size as reported in systables; it does not seem to matter what columns are used, specifically whether there are any variable length columns.
The formula for calculating rows per page is rows per page = trunc(pageuse/(rowsize + 4)) where pageuse is the page size in bytes minus 28 (source Informix manual, Estimating tables with fixed-length rows). We can use this to calculate the minimum row size that can be used for each page size without wasting space.
Page size (bytes) | Minimum row size to fill page based on formula | Smallest row size at which AUTOLOCATE will select this page size based on empirical evidence | Largest row size at which AUTOLOCATE will select this page size based on empirical evidence |
---|---|---|---|
2048 | 4 | 1 | 32074 |
4096 | 12 | 43 | 2021 |
6144 | 20 | 85 | 1515 |
8192 | 28 | 81 | 1615 |
10240 | 36 | 137 | 4591 |
12288 | 44 | 133 | 2403 |
14336 | 52 | 142 | 4574 |
16384 | 60 | 224 | 3187 |
For the largest row size, I didn’t exhaustively try every row size; these are just what is used in the database schema I loaded.
We can maybe begin to see what the engine is trying to do. There is clearly a smallest row size for which each page size will be chosen and this gives at least 2.7x head room, presumably to allow compression to be effective.
At the extreme, there is a seemingly rogue result for the table with a 32074 row size (comprised of a serial, int, varchar(64), char(32000) and char(1) columns) being put in a 2 kB dbspace. (I would have thought 16 kB would suit better, minimising the number of pages each row spans.) Some simple math(s) this suggests that minimising space may be the overriding consideration as the following examples show.
- 4096 page size: 2 x (2021 + 4) + 28 is 4078.
- 6144 page size: 4 x (1515 + 4) + 28 is 6104.
- 8192 page size: 5 x (1615 + 4) + 28 is 8123.
- 10240 page size: 2 x (4591 + 4) + 28 is 9218 (not such a good fit).
- 12288 page size: 5 x (2403 + 4) + 28 is 12035.
- 14336 page size: 3 x (4574 + 4) + 28 is 13762.
- 16384 page size: 5 x (3187 + 4) + 28 is 15973.
If we apply the same logic elsewhere what do we get? Looking at a range of ten values minimising the space wasted does indeed seem to be a goal.
Row size | Page size selected (kB) | Bytes wasted 2 kB | Bytes wasted 4 kB | Bytes wasted 6 kB | Bytes wasted 8 kB | Bytes wasted 10 kB | Bytes wasted 12 kB | Bytes wasted 14 kB | Bytes wasted 16 kB |
---|---|---|---|---|---|---|---|---|---|
133 | 12 | 102 | 95 | 88 | 81 | 74 | 67 | 60 | 53 |
134 | 8 | 88 | 66 | 44 | 22 | 0 | 116 | 94 | 72 |
135 | 6 | 74 | 37 | 0 | 102 | 65 | 28 | 130 | 93 |
136 | 4 | 60 | 8 | 96 | 44 | 132 | 80 | 28 | 116 |
137 | 10 | 46 | 120 | 53 | 127 | 60 | 134 | 67 | 0 |
138 | 6 | 32 | 92 | 10 | 70 | 130 | 48 | 108 | 26 |
139 | 8 | 18 | 64 | 110 | 13 | 59 | 105 | 8 | 54 |
140 | 2 | 4 | 36 | 68 | 100 | 132 | 20 | 52 | 84 |
141 | 4 | 135 | 8 | 26 | 44 | 62 | 80 | 98 | 116 |
142 | 14 | 122 | 126 | 130 | 134 | 138 | 142 | 0 | 4 |
I did these calculations in Excel so they must be right, yeah? Assuming some of the possible page sizes are just too large to be considered, the only thing this theory can’t explain is why for a 137 row size the engine chose a 10 kB page size over a 2 kB one.
What is clear is that using the largest page size possible, thus reducing the number of I/O requests is not a feature of AUTOLOCATE. Perhaps for many systems this doesn’t help anyway: in the OLTP world if you want to read a row and need to read in one 2 kB page or one 16 kB page to do so, which is actually more efficient on memory and the buffer pools?
Monitoring Informix with the Elastic Stack
Posted: 8 December, 2018 Filed under: Monitoring | Tags: elasticsearch, elk, informix, kibana, logstash Leave a commentIntroduction
If you’re not familiar with the Elastic Stack it is a suite of products for ingesting data or logs, searching, analysing and visualising. There is a good overview over at the Elastic web site of how it can be put together. I say “can” because the stack is very flexible and, for example, you can send JSON documents to Elasticsearch via a REST API, rather than use Filebeat or Logstash.
This blog post is mostly concerned with ingesting the Informix online log with Filebeat, recognising certain types of log line that can occur and tagging the file using rules set up in Logstash, before sending it to Elasticsearch for storage and indexing. Finally Kibana can be used to visualise the data stored in Elasticsearch.
It’s easy to see how this could be scaled up to provide a single place to visualise logs from multiple instances and it would be fairly trivial to add in other logs too, like the Informix bar logs and logs from the operating system.
At IIUG 2018 in Arlington, VA I presented a talk entitled DevOps for DBAs, which demonstrated the Docker set up now described below but I hadn’t at the time documented the full set up at the time. Here it is!
Practical demonstration with Docker containers
Overview
This demonstration sets up two containers: one running Informix Developer Edition and Filebeat to collect and ship logs:
- Informix 12.10.FC12W1DE, listening on port 9088/tcp for onsoctcp connections.
- Filebeat 6.5.2.
and the other running the Elasticstack components as follows:
- Logstash 6.5.2, listening on port 5044/tcp.
- Elasticsearch 6.5.2, listening on port 9200/tcp.
- Kibana 6.5.2, listening on port 5601/tcp.
Access to Kibana is via your favourite web browser running on your desktop. Nginx will be listening on port 80 so you can simply access http://localhost/
For a secure production implementation it’s recommended that you use Nginx with HTTPS as a reverse proxy for the Kibana web service as shown in the diagram. We’ll be using Nginx in this demonstration, rather than connecting to Kibana directly, but we won’t be configuring SSL; there are plenty of online guides about how to do this. Also communication between Filebeat and Logstash should be encrypted: this blog post doesn’t cover this.
The above versions are current at the time of writing (December 2018). Elasticstack moves very quickly so it is likely these will not be the latest versions by the time you read this. The idea of this blog post is that you should just be able to copy and paste the commands and end up with a working system but you shouldn’t be surprised if things don’t work perfectly if your versions don’t match the above. For example, in between beginning this blog post and finishing it, version 6.5.x was released with improved default security settings, with services only listening on the loopback interface without reconfiguration.
Running the whole Elasticstack in Docker plus Informix does require a reasonable amount of memory and I’d suggest a minimum of 2.5 GB to be allocated to the Docker Engine.
Docker network
To provide name resolution between the containers we are going to start by creating a docker network:
docker network create --driver bridge my_informix_elk_stack
Elastic Stack installation
Docker container
We’ll start by setting up the Elastic Stack Docker container which will be using on a (minimal) Debian installation. In a terminal run:
docker pull debian
docker run -it --name elasticstack_monitoring -p 80:80 -p 5044:5044 --hostname elasticstack -e "GF_SECURITY_ADMIN_PASSWORD=secret" --net my_informix_elk_stack debian
Your terminal should now be inside the Docker container and logged in as root.
To avoid some issues with debconf when installing packages run:
echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
Install Java
Run these commands to install some the software-properties-common package and then install OpenJDK 8, which is required by Elasticsearch and Logstash. Java 9 should be fine too.
The Debian Docker image does not come with many packages pre-installed so I am also going to install vim for editing files later plus a few other essentials; you may prefer nano or another editor.
apt-get update
apt-get install software-properties-common gnupg vim wget apt-transport-https openjdk-8-jre
The Debian Docker container is a basic installation so this short list of packages have hundreds of dependencies.
Check the Java version:
# java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Install Elasticsearch
The Elasticsearch installation is more straightforward than Oracle Java and follows standard Linux methods for setting up and installing from a third party software repository.
First we install the repository’s key:
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
Then add the repository:
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
We need the HTTPS transport package installing before we can proceed to install.
apt-get update
apt-get install elasticsearch
Elasticsearch will work right out of the box which is fine for the purposes of this demonstration and (after we start the service) will be listening on localhost only on ports 9200 for the REST API and 9300 for node communication.
Install Kibana
This is installed from the Elasticstack repository added above:
apt-get install kibana
Again Kibana doesn’t require any reconfiguration for the purposes of this demonstration and will listen on localhost only on port 5601.
Start Kibana by running:
service kibana start
Now that Kibana is running start Elasticsearch by running:
service elasticsearch start
It’s worth noting as this point that modern Debian distributions use systemd but this doesn’t work in non-privileged Docker containers. For reference the systemd equivalents for this are:
systemctl daemon-reload
systemctl enable kibana
systemctl enable elasticsearch
systemctl start kibana
systemctl start elasticsearch
This commands also ensure the service starts on boot.
As Kibana is only listening on localhost and therefore unreachable from an external web browser, we will set up Nginx as a reverse proxy. This is a more secure configuration and recommended for any production implementation because only Nginx is directly exposed to the internet and not Kibana.
Install Nginx as a reverse proxy
Start by installing Nginx:
apt-get install nginx
Edit the file /etc/nginx/sites-available/default and in the location / section add the line:
proxy_pass http://localhost:5601;
Comment out the line beginning with try_files.
It should look something like this:
location / { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. proxy_pass http://localhost:5601; #try_files $uri $uri/ =404; }
Don’t forget the semi-colon!
Start nginx:
service nginx start
Install Logstash
An installation procedure that by now should look familiar:
apt-get install logstash
Our Logstash configuration will be in two parts:
- A standard out of the box configuration for receiving files from Filebeat and sending them to Elasticsearch: for this copy /etc/logstash/logstash-sample.conf to /etc/logstash/conf.d/logstash.conf
- A custom config file, /etc/logstash/conf.d/informix.conf, for parsing the Informix online log.
I intend to update and improve the Logstash config for filtering the Informix online log and it’s available on my Github page at https://github.com/skybet/informix-helpers/blob/master/logstash/informix.conf. Download it locally and then you can copy it to your Docker container as follows:
docker cp informix.conf elasticstack_monitoring:/etc/logstash/conf.d/informix.conf
The Informix config file requires that Filebeat tags the file with “[field][informix] = true“; this condition is trivial to remove if you wish.
Check the Logstash configuration with:
/usr/share/logstash/bin/logstash -t --path.settings /etc/logstash
Finally start Logstash with:
/usr/share/logstash/bin/logstash --path.settings /etc/logstash
You could also use systemd to do this.
Informix installation
Informix Developer Edition Docker container
Now we are going to set up the Informix container to monitor. On your workstation in another terminal run:
$ docker pull ibmcom/informix-developer-database
docker run -it --name iif_developer_edition --privileged -p 9088:9088 -p 9089:9089 -p 27017:27017 -p 27018:27018 -p 27883:27883 --hostname ifxserver --net my_informix_elk_stack -e LICENSE=accept ibmcom/informix-developer-database:latest
It’s worth noting that if you exit the shell the Informix DE Docker container will stop. You can start it again with:
docker start iif_developer_edition -i
This latest version containing 12.10.FC12W1 doesn’t return you to the prompt after the engine starts, so you’ll need to open an interactive container in another terminal window as follows:
docker exec -it iif_developer_edition /bin/bash
Now both Docker containers are running you should be able to test name resolution and connectivity both ways with ping.
From the Informix container:
informix@ifxserver:/$ ping elasticstack_monitoring
From the Elastic Stack container:
root@elasticstack2:/# ping iif_developer_edition
These names belong to the containers and are not necessarily their host names.
While you’re still logged in as Informix set MSG_DATE to 1 which is required for my Logstash configuration:
onmode -wf MSG_DATE=1
This means (nearly) all online log messages will be prefixed with the date (MM/DD/YY format) and time (HH:MM:SS format).
You’ll be logged in as user informix which can sudo to root as follows:
sudo -i
Install Filebeat
In the Informix container it’s more of the same to install Filebeat:
apt-get update
apt-get install vim wget apt-transport-https
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-6.x.list
apt-get update
apt-get install filebeat
Filebeat’s configuration file is /etc/filebeat/filebeat.yml. If you’re not familiar with yml files correct indentation with two spaces per level is absolutely essential; yml files rely on this for structure instead of any braces or brackets. Lines beginning with a dash indicate an array.
onstat -c | grep MSGPATH reveals that the Informix online log file resides at /opt/ibm/data/logs/online.log and we want Filebeat to ingest this and pass it to Logstash running in the other container for parsing.
The Informix online log quite often contains wrapped lines and these generally don’t start with a timestamp or date of any kind.
Edit the file and add the following configuration directly under filebeat.inputs:
- type: log enabled: true paths: - /opt/ibm/data/logs/online.log fields: informix: true multiline.pattern: '^[0-9][0-9]' multiline.negate: true multiline.match: after
Finally set up the output by commenting out (add a ‘#’) to all parts of output.elasticsearch and uncommenting section output.logstash. Set hosts in this section to [“elasticstack_monitoring:5044”].
Start filebeat by running:
service filebeat start
You should see the message: Config OK.
Using Kibana
You should now be able to log into Kibana on http://localhost/ and add an index on filebeat*.
Then in the Discover section you should be able to see Informix log lines coming into the system and how they have been tagged.
If you wish to improve and test the parsing of particular log entries it is simple enough to create ones yourself in the Informix container like this:
echo "12/08/18 17:00:00 My Informix log line" >> /opt/ibm/data/logs/online.log
This particular blog post is going to end shortly. Kibana is a big subject and at this stage of my Elasticstack DevOps journey, I don’t feel qualified to write a blog on the subject. Once I’ve progressed further I may do a follow-on post.
The tagging of events in the online log like the checkpoint duration and its assignment to variable informix.ckpt_duration should allow you do easily do searches based on this and visualise them in dashboards.
Good luck!
Improving remote query performance by tuning FET_BUF_SIZE
Posted: 3 April, 2017 Filed under: SQL query tuning Leave a commentI thought I’d write blog post as a nice example of where tuning the client-side variable, FET_BUF_SIZE, really speeded up a remote query.
FET_BUF_SIZE is documented by IBM in the context of a Java application using JDBC here and as a server environment variable here.
One thing the documentation warns about is that simply setting this to a high value may degrade performance, especially if you have a lot of connections. With that in mind here are some facts about the query I’m running and using as a basis for these tests:
- I am just using a single connection to the database.
- the query returns around 10000 rows and 60 Mb of data.
- the client and the server are geographically separated from each other and Art Kagel’s dbping utility typically takes around 0.1 seconds to connect remotely; this compares with around 3 milliseconds locally.
- crucially the query runs in seconds locally on the server but takes over three minutes when run remotely.
If I begin running the query with the default value of FET_BUF_SIZE and monitor waits on the server, I can see that reads only go up slowly and that my session is waiting on a condition (indicated by the Y in position one of column two) more or less all the time:
> while [ 1 ] ; do
> onstat -u | grep thompson
> sleep 1
> done
Userthreads
address flags sessid user tty wait tout locks nreads nwrites
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 552 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 552 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 560 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 560 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 568 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 576 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 592 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 624 0
26eb492d18 Y--P-R- 76228 thompson 0 26e67cd298 0 0 624 0
The sixth column shows the rstcb value of the thread I’m waiting on. I can use onstat -g con (print conditions with waiters) to see that I’m waiting on the network:
> onstat -g con | grep -E '^cid|26e67cd298'
cid addr name waiter waittime
5789 26e67cd298 netnorm 84353 0
A quick check with onstat -g ses 76228 shows that thread id. 84353 does indeed correspond to my session.
While the wait time shown above is not increasing it’s a different story when we look at netstat, again on the server:
> netstat -nc | grep '172.16.0.1'
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1312 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1284 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1306 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1302 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1194 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1206 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1266 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1304 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1318 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
tcp 0 1248 10.0.0.1:9088 172.16.0.1:37004 ESTABLISHED
What the above is showing us is that there are consistently around 1200 to 1300 bytes in the send queue (Send-Q). This is surely our bottleneck.
At this point when investigating the problem I considered modifying other parameters such as OPTOFC and Linux kernel parameters. However with a few moment’s thought it was clear these weren’t going to gain anything: OPTOFC optimises the open-fetch-close sequence and for a single long running query this is not going to give us anything measurable; and an investigation into increasing the Linux kernel parameter related to the send queue size was dismissed when we found that 1300 bytes was well below the maximum allowed.
In Informix 11.50 the maximum value of FET_BUF_SIZE is 32767 (32 kb) but this is increased to 2147483648, or as we’ll see actually 2147483647, (2 Gb) in 11.70 and above. We can therefore move onto to experiment with different values:
FET_BUF_SIZE | Query run time (s) | Average Send-Q size over 10 samples | Maximum Send-Q size observed |
---|---|---|---|
Default | 221.2 | 1274 | 1332 |
1024 | 221.1 | 1255 | 1326 |
2048 | 221.1 | 1285 | 1338 |
4096 | 221.2 | 1297 | 1360 |
6144 | 102.1 | 2564 | 2676 |
8192 | 56.6 | 5031 | 5210 |
16384 | 22.6 | 12490 | 13054 |
32767 (max. 11.50 value) | 11.5 | 24665 | 29968 |
65536 | 7.0 | 62188 | 62612 |
131072 | 4.9 | 115793 | 127826 |
262144 | 4.0 | 146686 | 237568 |
524288 | 3.5 | 184320 | 249856 |
1048576 | 3.3 | 245760 | 473616 |
2097152 | 3.2 | 249856 | 486352 |
2147483647 (max. value – 1) | 3.0 | 245760 | 549352 |
2147483648 (supposed max. value) | 221.3 | 1276 | 1366 |
As the run times get shorter it gets tricky to measure the Send-Q using netstat -nc: it can be sampled very frequently using a command like:
while [ 1 ] ; do
netstat -n | grep '172.16.0.1'
done
This will produce many measurements per second and with this it’s possible to see it fill up and drain several times in the period while the statement is running.
It’s also interesting to play around with the boundaries. For example, with a FET_BUF_SIZE between around 5500 and 5600 maximum Send-Q sizes the same as those consistently achieved with a FET_BUF_SIZE of 6144 begin to creep into the results but many measurements remain around the values consistently measured wit a FET_BUF_SIZE of 4096:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 1316 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1318 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1278 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1352 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1288 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 2546 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1278 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 2502 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1266 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1314 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 2506 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
tcp 0 1292 10.0.0.1:9088 172.16.0.1:37488 ESTABLISHED
So what are the conclusions?
- Increasing FET_BUF_SIZE at the client side can dramatically improve the speed of remote queries.
- Maximum Send-Q sizes, as measured by netstat, increase in discrete steps as FET_BUF_SIZE is increased.
- A larger Send-Q allows more data to be cached and reduces waits seen in Informix.
- To see any improvement at all FET_BUF_SIZE must be increased to at least 6000 (approximate value).
- Around boundaries between maximum Send-Q sizes there appears to be a cross-over region where maximum send queue sizes overlap from two adjacent values are seen from one second to the next.
- The maximum value allowed in 11.70 at least is 2147483647 and not 2147483648, as indicated in the documentation.
- The maximum 11.50 value of 32767 produced a run time nearly 4x slower than an optimised value for 11.70+
- Other testing I did, not documented here, shows that the results are uniform across JDBC and ESQL/C applications.
Note: all user names, IP addresses and port numbers used in this post have been altered.
Informix or Client SDK install: No Java virtual machine could be found
Posted: 5 January, 2017 Filed under: Uncategorized 1 CommentThis is a something of a note to self. For some time it has been been the case that you may see this message when attempting an Informix server or Client SDK install if there is a problem starting the installer’s Java runtime environment:
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:/lib64 ./ids_install
Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
No Java virtual machine could be found from your PATH
environment variable. You must install a VM prior to
running this program.
To add insult to injury when this condition occurs the installer exits with status code zero, suggesting all is ok.
Now the obvious thing to do seems to be to install a Java package, wondering whether OpenJDK will suffice or the official Oracle version is needed. This is never the answer! The Informix installer comes bundled with its own Java run time environment (JRE) which gets extracted into /tmp/install.dir.X and your challenge is in fact to find out why it isn’t working as it should.
You can see in my attempt at installing the product I have already prefaced the command with LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:/lib64
. This is already a known way of fixing some installation problems. (For a 32-bit version you’d simply use /usr/lib:/lib
.)
Everyone’s friend, strace, is a great way to start investigating this problem. In amongst the output I find this:
faccessat(AT_FDCWD, "/tmp/install.dir.12813/Linux/resource/jre/jre/bin/java", X_OK) = -1 EACCES (Permission denied)
So why is this? I am logged in as root so I ought not be running into permission denied issues.
The core problem here is the way /tmp, which is a separate filesystem on my machine, is mounted. From the mount command output:
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)
The key part here is the noexec flag which is a security feature preventing execution of binary files residing on this filesystem.
The best way to fix this is to set the environment variable IATEMPDIR to a directory on a filesystem where execution is allowed. I usually use /root for this purpose. And success:
# export IATEMPDIR=/root
# LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64:/lib64 ./ids_install
Preparing to install...
Extracting the JRE from the installer archive...
Unpacking the JRE...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
Launching installer...
Preparing CONSOLE Mode Installation...
While the above should be sufficient I have seen the server installer still fail to work even with this environment variable set as some files may still be placed in /tmp. In this situation you can temporarily remove the security restriction with:
mount -o remount,rw,nosuid,nodev,relatime,exec /tmp
and switch it back on again with:
mount -o remount,rw,nosuid,nodev,relatime,noexec /tmp
I suggest before running the above you check the existing mount options for your /tmp filesystem.
Intermittent “CSM: authentication error” with JDBC
Posted: 21 December, 2016 Filed under: Compliance, SQL and SPL debugging 1 CommentThis article will only concern you if:
- you connect using JDBC.
- you use simple password encryption.
Simple password encryption just does one thing: it encrypts the password sent to the database server in transit preventing it from being obtainable by network packet sniffing.
At the client end configuring simple password encryption can be done simply by adding
";SECURITY=PASSWORD"
to your JDBC connection string.
At the server end set up a DBSERVERALIAS and add
csm=(SPWDCSM)
to the fifth field in sqlhosts and set environment variable INFORMIXCONCSMCFG before starting the instance to point to a file containing something like:
SPWDCSM("/opt/informix/lib/csm/libixspw.so", "", "p=1")
This is covered in more detail elsewhere and I haven’t covered using CSDK but for JDBC connections it’s all there is to know.
Unfortunately there is a bug in JDBC 3.70.JC8W1 and JDBC 4.10.JC7 and below where every 100th connection attempt or so will fail randomly with this stack:
java.sql.SQLException: CSM: authentication error.
at com.informix.jdbc.IfxSqliConnect.(IfxSqliConnect.java:1337)
at sun.reflect.GeneratedConstructorAccessor3.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.informix.jdbc.IfxDriver.connect(IfxDriver.java:243)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at Connect.main(Connect.java:30)
Caused by: com.informix.asf.IfxASFRemoteException:
at com.informix.asf.Connection.recvBindResponse(Connection.java:1363)
at com.informix.asf.Connection.establishConnection(Connection.java:1619)
at com.informix.asf.Connection.(Connection.java:392)
at com.informix.jdbc.IfxSqliConnect.(IfxSqliConnect.java:1232)
... 7 more
You can see if you’re vulnerable by compiling this app and running it until it fails or you’re reasonably confident you don’t have a problem:
public class Connect
{
public static void main(String[] args)
{
Connection conn = null;
String url = "jdbc:informix-sqli://hostname:port/dbname:INFORMIXSERVER=informixserver;user=user;password=password;SECURITY=PASSWORD";
System.out.println(url);
try
{
Class.forName("com.informix.jdbc.IfxDriver");
}
catch (Exception e)
{
System.out.println("FAILED to load Informix JDBC driver.");
e.printStackTrace();
return;
}
int i=0;
while (true) {
i++;
try
{
conn = DriverManager.getConnection(url);
}
catch (SQLException e)
{
System.out.println("FAILED to connect! "+e);
e.printStackTrace();
}
System.out.println("Connected " + i);
if (conn != null) {
try {
conn.close();
}
catch (SQLException e) {
System.out.println("FAILED to disconnect! "+e);
e.printStackTrace();
}
}
}
}
}
If your application handles fails connections and retries automatically you might not have noticed this error or perhaps it was lost in the noise but for more simple applications it can be a pain.
Fortunately this is fixed in JDBC 4.10.JC8 and the fix works with 11.70 and 12.10 versions of the servers.
Interestingly the JDBC release notes for 4.10.JC8 are coy about this, showing just one fix.