using RunCommand to wrap LXD linux container crashes

[nomorelogic]

hello everyone

I have a problem using RunCommand (unit process); this is a special case as I am doing some testing with linux LXD containers.

My system is DEVUAN:

Code: [Select]

uname -a
Linux myhost 6.1.0-30-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.124-1 (2025-01-12) x86_64 GNU/Linux

LXD version:

Code: [Select]

$ lxc --version
5.0.2
.

The program is trivial and, once compiled, it must be launched from the shell as root user or using "sudo":

Code: [Select]

sudo ./testlxc.

Below is the code.

Code: [Select]

program testlxc;
uses process;

const
  LXC_BIN = ‘/usr/bin/lxc’;

var AStdOut: string;
begin

  Writeln(LXC_BIN + ‘ LIST ’);
  RunCommand(LXC_BIN, [‘list’], AStdOut); // this works
  Writeln(‘output: >>’);
  Writeln(AStdOut);
  Writeln(‘<<’);


  Writeln(LXC_BIN + ‘ INIT ’);
  RunCommand(LXC_BIN, [‘init’, ‘ubuntu:20.04’, ‘u1’], AStdOut); // here it crashes
  Writeln(‘output: >>’);
  Writeln(AStdOut);
  Writeln(‘<<’);

end.

.

The first invocation of RunCommand with parameter ‘list’ works correctly as expected.
The second invocation with parameter ‘init’, on the other hand, crashes.

The result is as follows:

Code: [Select]

$ sudo ./testlxc
/usr/bin/lxc LIST
output: >>
+------+---------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| m1 | STOPPED | | CONTAINER | 0 |
+------+---------+------+------+-----------+-----------+

<<
/usr/bin/lxc INIT

Running from terminal:

Code: [Select]

sudo lxc list.
or

Code: [Select]

sudo lxc init ubuntu:20.04 u1everything works correctly and smoothly.

I can't see where the problem lies.
Any ideas?

thanks
nomorelogic

[MarkMLl]

I'd suggest that the first thing to try would be running it as root rather than using sudo.

If that fails in the same way I'd suggest supplying a couple of dummy parameters in the first RunCommand() invocation, so that you can be sure that you're comparing like with like.

I've not used LXD, but from my experience with Docker etc. I think you need to focus on whether it's RunCommand() which is crashing (i.e. before transferring control into lxc), or lxc (after control has been transferred into it) because e.g. it wants some resource which it still can't get at despite the use of sudo.

MarkMLl

[nomorelogic]

thanks for your suggestions

I ran the test as root user, unfortunately not much changed
I also added some parameters to the first invocation of RunCommand

Code: Pascal  [Select][+][-]

1. program testlxc;
2. uses process;
3.  
4. const
5.   LXC_BIN = '/usr/bin/lxc';
6.  
7. var AStdOut: string;
8. begin
9.  
10.   Writeln(LXC_BIN + ' LIST ');
11.   RunCommand(LXC_BIN, ['list', '-f', 'json'], AStdOut);   // <- works
12.   Writeln('output: >>');
13.   Writeln(AStdOut);
14.   Writeln('<<');
15.  
16.  
17.   Writeln(LXC_BIN + ' INIT ');
18.   RunCommand(LXC_BIN, ['init', 'ubuntu:20.04', 'u1'], AStdOut);    // <- crashes
19.   Writeln('output: >>');
20.   Writeln(AStdOut);
21.   Writeln('<<');
22.  
23. end.
24.  

this is the output (executed as root)

Code: Pascal  [Select][+][-]

1. # ./testlxc
2. /usr/bin/lxc LIST
3. output: >>
4. [{"architecture":"x86_64","config":{"image.architecture":"amd64","image.description":"Macaroni OS Eagle Minimal amd64 (20250127_15:11)","image.os":"Macaroni OS","image.release":"current","image.serial":"20250127_15:11","image.type":"squashfs","volatile.base_image":"00e40cbff9f6be5a7524cc3ee64cc5bafe28b85e8a94cf72193450af30f2d2f5","volatile.cloud-init.instance-id":"ce50a30b-7add-4ecd-aee5-be0d7b7a0e50","volatile.eth0.hwaddr":"00:16:3e:e1:1f:8d","volatile.idmap.base":"0","volatile.idmap.current":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.idmap.next":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.last_state.idmap":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.last_state.power":"STOPPED","volatile.uuid":"d134e8c3-9f00-4b14-a863-a7a393e21e13"},"devices":{"sharedsrc":{"path":"/tmp","source":"/media/dati/dev/lxc/test001/","type":"disk"}},"ephemeral":false,"profiles":["default"],"stateful":false,"description":"","created_at":"2025-01-30T14:44:59.593127664Z","expanded_config":{"image.architecture":"amd64","image.description":"Macaroni OS Eagle Minimal amd64 (20250127_15:11)","image.os":"Macaroni OS","image.release":"current","image.serial":"20250127_15:11","image.type":"squashfs","volatile.base_image":"00e40cbff9f6be5a7524cc3ee64cc5bafe28b85e8a94cf72193450af30f2d2f5","volatile.cloud-init.instance-id":"ce50a30b-7add-4ecd-aee5-be0d7b7a0e50","volatile.eth0.hwaddr":"00:16:3e:e1:1f:8d","volatile.idmap.base":"0","volatile.idmap.current":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.idmap.next":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.last_state.idmap":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":10000001}]","volatile.last_state.power":"STOPPED","volatile.uuid":"d134e8c3-9f00-4b14-a863-a7a393e21e13"},"expanded_devices":{"eth0":{"name":"eth0","network":"lxdbr0","type":"nic"},"root":{"path":"/","pool":"default","type":"disk"},"sharedsrc":{"path":"/tmp","source":"/media/dati/dev/lxc/test001/","type":"disk"}},"name":"m1","status":"Stopped","status_code":102,"last_used_at":"2025-02-03T16:35:51.011958116Z","location":"none","type":"container","project":"default","backups":null,"state":{"status":"Stopped","status_code":102,"disk":{"root":{"usage":169681408}},"memory":{"usage":0,"usage_peak":0,"swap_usage":0,"swap_usage_peak":0},"network":null,"pid":0,"processes":0,"cpu":{"usage":0}},"snapshots":null}]
5.  
6. <<
7. /usr/bin/lxc INIT
8.  

probably now I have to figure out if RunCommand crashes or if lxc is waiting for some resource.
The only problem I have is that I cannot debug with Lazarus as I cannot launch Lazarus as root.

[MarkMLl]

The only problem I have is that I cannot debug with Lazarus as I cannot launch Lazarus as root.

That tends to depend on the widget set, and version thereof. However a reliable workaround is to start the program using gdbserver, and attach to that using Lazarus.

MarkMLl

[nomorelogic]

working using gdb at command line I got this

Code: Pascal  [Select][+][-]

1. (gdb) run
2. Starting program: /media/dev/lxc/testlxc
3. /usr/bin/lxc LIST
4. [Detaching after fork from child process 5888]
5. output: >>
6. [{"architecture":"x86_64",  [...cut...]  ,"snapshots":null}]
7.  
8. <<
9. /usr/bin/lxc INIT
10. [Detaching after fork from child process 5895]
11.  

I was reflecting on the gdb message before the crash
Detaching after fork from child process 5895

Looks like RunCommand did its job.

[MarkMLl]

Note that I said gdbserver, not gdb.

Looks like RunCommand did its job.

Maybe. Can you see the program's parameters etc. in its /proc entry?

MarkMLl

[nomorelogic]

I guess I don't know how to use gdbserver now... 

in /proc I find this

Code: Pascal  [Select][+][-]

1. root@myhost:/proc/7681# cat cmdline
2. /usr/bin/lxcinitubuntu:20.04u1
3.  

I do not know whether it is normal that in cmdline the arguments do not have the space as separator.
However the command is present in proc.


Edit:

this is the output of "lxc monitor" until the crash

Code: Pascal  [Select][+][-]

1. # lxc monitor --type=logging --pretty
2. DEBUG  [2025-02-07T09:18:42+01:00] Event listener server handler started         id=0291e421-5d89-4bb2-8652-c739d874f13f local=/var/lib/lxd/unix.socket remote=@
3. DEBUG  [2025-02-07T09:18:46+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
4. DEBUG  [2025-02-07T09:18:46+01:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/instances?filter=&recursion=2" username=root
5. DEBUG  [2025-02-07T09:18:46+01:00] GetInstanceUsage started                      instance=m1 project=default
6. DEBUG  [2025-02-07T09:18:46+01:00] GetInstanceUsage finished                     instance=m1 project=default
7. DEBUG  [2025-02-07T09:19:05+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
8. DEBUG  [2025-02-07T09:19:05+01:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/instances?filter=&recursion=2" username=root
9. DEBUG  [2025-02-07T09:19:05+01:00] GetInstanceUsage started                      instance=m1 project=default
10. DEBUG  [2025-02-07T09:19:05+01:00] GetInstanceUsage finished                     instance=m1 project=default
11.  

[MarkMLl]

I guess I don't know how to use gdbserver now... 

Allowing that you've probably moved from the problem being inside the Pascal program to its being inside the one you're invoking, it might not be directly relevant. However I felt that I did have to draw your attention to the distinction, because it's what you need it you're trying to debug something that either runs as root or has to have POSIX capabilities set (which is a much more controlled way of doing things).

in /proc I find this

Code: Pascal  [Select][+][-]

1. root@myhost:/proc/7681# cat cmdline
2. /usr/bin/lxcinitubuntu:20.04u1
3.  

I do not know whether it is normal that in cmdline the arguments do not have the space as separator.

Try piping the output through xxd, since cat won't show embedded /0 characters which might be being used as the delimiter.

MarkMLl

[nomorelogic]

thank you for your suggestions
I will try to learn more about dbgserver

launch parameters seems corrects using xxd

Code: Pascal  [Select][+][-]

1. # xxd /prov/16065/cmdline
2. 00000000: 2f75 7372 2f62 696e 2f6c 7863 0069 6e69  /usr/bin/lxc.ini
3. 00000010: 7400 7562 756e 7475 3a32 302e 3034 0075  t.ubuntu:20.04.u
4. 00000020: 3100 2d2d 6465 6275 6700                 1.--debug.
5.  

I'm thinking that, as soon as possible, I'll look into some lxd forums
I will update here if I find anything useful

[MarkMLl]

I will try to learn more about dbgserver

gdbserver.

OK, so you can see /0 separators so things are getting that far.

MarkMLl

[nomorelogic]

I'm thinking that, as soon as possible, I'll look into some lxd forums

I prepared a simple c source with the intention of asking for help in some lxd forum
well....
this works 

Code: C  [Select][+][-]

1. #include <stdlib.h>
2. #include <stdio.h>
3.  
4. int main() {
5.   int status = 0;
6.  
7.   printf("lxc LIST\n");
8.   printf("Output >> \n");
9.   status = system("/usr/bin/lxc list -f json");
10.   printf("<< (Status=%d)\n\n", status);
11.  
12.   printf("lxc INIT\n");
13.   printf("Output >> \n");
14.   status = system("/usr/bin/lxc init ubuntu:20.04 u1");
15.   printf("<< (Status=%d)\n");
16.  
17.   return 0;
18. }
19.  

this is the correct output

Code: Pascal  [Select][+][-]

1. sudo ./testlxc
2. lxc LIST
3. Output >>
4. [{"architecture":"x86_64", ...[cut] ...,"snapshots":null}]
5. << (Status=0)
6.  
7. lxc INIT
8. Output >>
9. Creating u1
10. << (Status=-1768918016)
11.  

I'm starting to get a little confused.
I will investigate further.

[MarkMLl]

Gut feeling is that it's something to do with RunCommand(), e.g. the shell/environment variables being corrupted.

I wonder what would happen if you replaced the two invocations of RunCommand() with invocations of distinct TProcess instances? My rationale there is that in principle at least there will be no retained state between the instances... it's what I tend to use and (touch wood) I've never had any inexplicable problems.

MarkMLl

[MarkMLl]

I'm a bit uncomfortable here: I was considering loading up what you've got onto a machine so that I could write it up for colleagues, but I notice from https://en.wikipedia.org/wiki/LXC that LXC and LXD appear to be distinct: can you tell me exactly what Debian-style packages you installed to get to your current position?

MarkMLl

[nomorelogic]

can you tell me exactly what Debian-style packages you installed to get to your current position?

Of course.
I first installed LXC and then LXD and I did both installations via Synaptic.
I am not sure if it is necessary to install LXC but this is what I did in my case.

For LXC:

• lxc
• lxc-templates
• lxcfs
• liblxc-common
• liblxc1
• libvirt-daemon-driver-lxc

For LXD (subsequent installation)

• lxd
• lxd-agent
• lxd-client

After LXD installation, I ran ‘lxd init’, confirming with enter each of the proposed defaults.

It is correct to say that LXC and LXD are different, although the latter (LXD) rests on LXC.
To easily distinguish the 2 systems:
LXD: for image management, there is a single command  called ‘lxc’ which accepts parameters (the one I am using now in my tests)
LXC: for image management, it has a series of commands starting with ‘lxc-’, such as ‘lxc-create’, ‘lxc-attach’ and so on

Thank you
nomorelogic

[nomorelogic]

I wonder what would happen if you replaced the two invocations of RunCommand() with invocations of distinct TProcess instances?

I did the test you suggested and using 2 successive invocations of TProcess works!
This is the code.

Code: Pascal  [Select][+][-]

1. program testlxc2;
2. uses process;
3.  
4. const
5.   LXC_BIN = '/usr/bin/lxc';
6.  
7. var AStdOut: string;
8.   AProcess: TProcess;
9. begin
10.  
11.   Writeln(LXC_BIN + ' LIST using TProcess');
12.   AProcess := TProcess.Create(nil);
13.   try
14.     AProcess.Executable:= LXC_BIN;
15.     AProcess.Parameters.Add('list');
16.     AProcess.Parameters.Add('-f');
17.     AProcess.Parameters.Add('json');
18.     AProcess.Options := AProcess.Options + [poWaitOnExit];
19.     Writeln('Output >>');
20.     AProcess.Execute;
21.  
22.   finally
23.     AProcess.Free;
24.     Writeln('<< Output of "LIST"');
25.   end;
26.  
27.   Writeln('');
28.  
29.   Writeln(LXC_BIN + ' INIT using TProcess');
30.   AProcess := TProcess.Create(nil);
31.   try
32.     AProcess.Executable:= LXC_BIN;
33.     AProcess.Parameters.Add('init');
34.     AProcess.Parameters.Add('ubuntu:20.04');
35.     AProcess.Parameters.Add('u1');
36.     AProcess.Options := AProcess.Options + [poWaitOnExit];
37.     Writeln('Output >>');
38.     AProcess.Execute;
39.  
40.   finally
41.     AProcess.Free;
42.     Writeln('<< Output of "INIT"');
43.   end;
44.  
45.   Writeln('');
46.  
47. end.
48.  

At this point I would say that, at least in this case, there is a bug in RunCommand.