Run soltest and cli tests on Linux ARM

[matheusaaguiar]

Currently we have ARM releases but we only run bytecode equivalence tests.
This PR adds a job in the CI to run soltest and cli tests on ARM binary.
Closes #11351.
Closes #16321.

depends on #16328.

[cameel]

I thought you went to sleep already so I actually also started a quick PR for this :) #16321

I got through yaml syntax errors, but still not sure if the tests will actually pass. If you're working on it now, please take a look and feel free to pull the missing parts into yours if you need them. I'll just focus on other things I wanted to do in the meantime.

[matheusaaguiar]

I thought you went to sleep already so I actually also started a quick PR for this :) #16321

I also thought you went to sleep and didn't check to see if there was already work in that direction :D

I got through yaml syntax errors, but still not sure if the tests will actually pass. If you're working on it now, please take a look and feel free to pull the missing parts into yours if you need them. I'll just focus on other things I wanted to do in the meantime.

Ok, will do.

[cameel]

CLI tests fail with:

Incorrect output on stderr received. Expected:
Error: "file_not_found.sol" is not found.
But got:
test/cmdlineTests.sh: line 208: /root/project/build/solc/solc: No such file or directory

I'd check if the binary the build job saves to the workspace is under the same path as in the b_ubu job. I suspect it's not.

The soltest job is probably failing for a similar reason.

[matheusaaguiar]

I'd check if the binary the build job saves to the workspace is under the same path as in the b_ubu job. I suspect it's not.

The soltest job is probably failing for a similar reason.

Yes, this is the reason. The build job saves the binaries with prefix linux-arm.
I will try to see if I can find a good way of adjusting for that.

[cameel]

Nice solution. We probably should do it with macOS as well (not necessarily in this PR). There we instead use run_soltest (which is an internal detail of the soltest command) because the executable name is different.

[cameel]

If the CI passes this is good enough, though it could also use a few small tweaks (see my comments). If you want to merge this already, they could be done in a separate PR as well.

[cameel]

Argh, it actually did fail.

[nikola-matic]

There's no libevmone.so in /usr/local/lib at all - no mention of evmone anywhere on the VM in fact. It could be an issue with the actual container.

I SSH-ed into the failed job VM earlier today and installed it manually by following every step from the dockerfile, but that did indeed work and installed evmone properly.

[nikola-matic]

Ah, looks like the buildpack dep is the culprit - we're not copying /usr/local/ from libraries into the final build.

[matheusaaguiar]

There are 3 smt tests which are failing:
smtCheckerTests/external_calls/external_call_indirect_2.sol
smtCheckerTests/external_calls/external_call_indirect_5.sol
smtCheckerTests/functions/functions_external_2.sol

The message changes from "violations happens" to "violation might happen", e.g. (smtCheckerTests/external_calls/external_call_indirect_2.sol):

Expected result:
  Warning 6328: (434-455): CHC: Assertion violation happens here.
  Warning 6328: (1270-1291): CHC: Assertion violation happens here.
  Info 1391: CHC: 2 verification condition(s) proved safe! Enable the model checker option "show proved safe" to see all of them.
Obtained result:
  Warning 6328: (434-455): CHC: Assertion violation happens here.
  Warning 6328: (1270-1291): CHC: Assertion violation might happen here.
  Info 1391: CHC: 2 verification condition(s) proved safe! Enable the model checker option "show proved safe" to see all of them.

Not sure if this happens because of ARM.
@blishko, can you take a look, please? What do you think?

[blishko]

What I see now is that soltest is failing because it could not load evmone: libevmone.so: cannot open shared object file: No such file or directory.

I would need to see the failures.
It's possible that z3 behaves differently on arm.
Let me know when you fix the missing evmone problem.

[blishko]

OK, I see. I'll take a look at those tests tomorrow.

[blishko]

@matheusaaguiar : #16330 is my proposed solution

[nikola-matic]

@matheusaaguiar : #16330 is my proposed solution

Rebasing now, let's hope it helps.

[nikola-matic]

@matheusaaguiar : #16330 is my proposed solution

Looks like that did it - but bytecode compare is not unfortunately failing - luckily - not because of the actual bytecode comparison, but rather due to the run files persisted having the same name. @cameel do you have any ideas, as I can't really see anything wrong in config.yml?

[cameel]

@cameel do you have any ideas, as I can't really see anything wrong in config.yml?

Is it still an issue or did you solve it? I see all the jobs passing now in CI.

[nikola-matic]

@cameel do you have any ideas, as I can't really see anything wrong in config.yml?

Is it still an issue or did you solve it? I see all the jobs passing now in CI.

Oh nice, @matheusaaguiar fixed it. What did you do?

[cameel]

Any reason #16322 (comment) was reverted?

[matheusaaguiar]

@cameel, We were getting this error (which @nikola-matic mentioned) when running bytecode compare:

Applying workspace layers
  0fff5a7b-a1ab-40c2-bf18-2531188c1769
  e6fd50ab-8077-45f8-b97b-3715cef3635b
  4cc2f52f-5512-4ef9-908b-81e6028a1956
  588dd45a-fe9d-4c8f-83d7-3e28f21a8657
Concurrent upstream jobs persisted the same file(s) into the workspace:
  - test/tools/solfuzzer
  - test/soltest

Error applying workspace layer for job 588dd45a-fe9d-4c8f-83d7-3e28f21a8657: Concurrent upstream jobs persisted the same file(s)

So I reverted #16322 (comment) and updated the test jobs to use more fine-grained steps (instead of calling the templates run_soltest and run_cmdline_tests), so we can make symbolic links with the expected binaries names before running soltest and cli tests.