Backend TF not loaded

[Akhtar303]

Hi @itamarhaber @MeirShpilraien
I have machine which has no SSE4.1 SSE4.2 AVX AVX2 that's why Backend TF not loaded.
go to gear environment and import tensorflow but not import Tensorflow succesfully
i.e
(python3) onstak@onstak:/opt/redislabs/lib/modules/python3$

import tensorflow
Illegal instruction (core dumped)
and when I ran init.py tensorflow not load successfully but torch load successfully
Error
Backend TF not loaded, will try loading default backend
/home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7fcfdcf4fa59]
Symbol: (null) (base: (nil))
Module: /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1 (base 0x7fcfdc82a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin

but torch load successfully
TORCH backend loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_torch/redisai_torch.so
I am also try this one go to /opt/redislabs/lib/modules/python3$ and install tesorflow like pipenv install tensorflow or

sudo pipenv install tensorflow-1.14.0-cp37-cp37m-linux_x86_64.whl
but facing same core dump error
so kindly tell me how to solve this problem.
Thanks

[Akhtar303]

@itamarhaber Thanks for reply
I am working on EdgeRealtimeVideoAnalytics
link = https://github.com/RedisGears/EdgeRealtimeVideoAnalytics

when start redis with redisgear there is no error but when I ran python init.py then Tensorflow didn't load successfully but pytorch load successfully
Eroor
80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend
redis broke with this error

tting after a reboot. Redis must be restarted after THP is disabled.
80544:M 09 Oct 2019 08:39:51.745 * Module 'ai' loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so
80544:M 09 Oct 2019 08:39:51.844 * RedisGears version 99.99.99, git_sha=5beab40cb5e51e66a4b08cf2cabaf9155e9d8f89
80544:M 09 Oct 2019 08:39:51.844 * PythonHomeDir:/opt/redislabs/lib/modules/python3/
80544:M 09 Oct 2019 08:39:51.844 * MaxExecutions:1000
80544:M 09 Oct 2019 08:39:51.844 * ProfileExecutions:0
80544:M 09 Oct 2019 08:39:51.844 * PythonAttemptTraceback:1
80544:M 09 Oct 2019 08:39:51.844 * RedisAI api loaded successfully.
could not initialize RediSearch_CheckApiVersionCompatibility
80544:M 09 Oct 2019 08:39:51.844 # could not initialize RediSearch api, running without Search support.
80544:M 09 Oct 2019 08:39:51.982 * Initializing Python environment with: exec(open('/opt/redislabs/lib/modules/python3/.venv/bin/activate_this.py').read(), {'file': '/opt/redislabs/lib/modules/python3/.venv/bin/activate_this.py'})
80544:M 09 Oct 2019 08:39:52.256 * Module 'rg' loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisGears/redisgears.so
80544:M 09 Oct 2019 08:39:52.256 * Ready to accept connections
80544:M 09 Oct 2019 08:40:45.719 # Backend TORCH not loaded, will try loading default backend

80544:M 09 Oct 2019 08:40:46.930 * TORCH backend loaded from /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_torch/redisai_torch.so
80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend

=== REDIS BUG REPORT START: Cut & paste starting from here ===
80544:M 09 Oct 2019 08:40:50.828 # Redis 5.0.6 crashed by signal: 4
80544:M 09 Oct 2019 08:40:50.828 # Crashed running the instruction at: 0x7ff15bb4fa59
80544:M 09 Oct 2019 08:40:50.828 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7ff15bb4fa59]

Backtrace:
redis-server *:6379(logStackTrace+0x5a)[0x56095f9424da]
redis-server *:6379(sigsegvHandler+0xb1)[0x56095f942c91]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7ff1a7df2890]
/home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1(+0x725a59)[0x7ff15bb4fa59]
/lib64/ld-linux-x86-64.so.2(+0x10733)[0x7ff1a87b9733]
/lib64/ld-linux-x86-64.so.2(+0x151ff)[0x7ff1a87be1ff]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f)[0x7ff1a7b562df]
/lib64/ld-linux-x86-64.so.2(+0x147ca)[0x7ff1a87bd7ca]
/lib/x86_64-linux-gnu/libdl.so.2(+0xf96)[0x7ff1a8207f96]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x6f)[0x7ff1a7b562df]
/lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x2f)[0x7ff1a7b5636f]
/lib/x86_64-linux-gnu/libdl.so.2(+0x1735)[0x7ff1a8208735]
/lib/x86_64-linux-gnu/libdl.so.2(dlopen+0x71)[0x7ff1a8208051]
/home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so(RAI_LoadBackend_TensorFlow+0x60)[0x7ff1a55ee0c0]
/home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/redisai.so(RedisAI_ModelSet_RedisCommand+0x369)[0x7ff1a55eba59]
redis-server *:6379(RedisModuleCommandDispatcher+0x68)[0x56095f9702a8]
redis-server *:6379(call+0xb5)[0x56095f8fbc35]
redis-server *:6379(processCommand+0x36f)[0x56095f8fc33f]
redis-server *:6379(processInputBuffer+0x171)[0x56095f90ce31]
redis-server *:6379(aeProcessEvents+0x150)[0x56095f8f5670]
redis-server *:6379(aeMain+0x2b)[0x56095f8f5a5b]
redis-server *:6379(main+0x4f4)[0x56095f8f2604]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7ff1a7a10b97]
redis-server *:6379(_start+0x2a)[0x56095f8f285a]

------ INFO OUTPUT ------

Server

redis_version:5.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9260170b247e88b
redis_mode:standalone
os:Linux 4.15.0-65-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.4.0
process_id:80544
run_id:972bf07279b81aac3e38a2dba32475cf2e576eba
tcp_port:6379
uptime_in_seconds:59
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:10329362
executable:/home/onstak/EdgeRealtimeVideoAnalytics/redis-server
config_file:/home/onstak/EdgeRealtimeVideoAnalytics/redisedge/redisedge.conf

Clients

connected_clients:2
client_recent_max_input_buffer:198311946
client_recent_max_output_buffer:0
blocked_clients:0

Memory

used_memory:544659192
used_memory_human:519.43M
used_memory_rss:258641920
used_memory_rss_human:246.66M
used_memory_peak:544659192
used_memory_peak_human:519.43M
used_memory_peak_perc:197.18%
used_memory_overhead:250158447
used_memory_startup:791728
used_memory_dataset:294500745
used_memory_dataset_perc:54.15%
allocator_allocated:209405176
allocator_active:209842176
allocator_resident:284135424
total_system_memory:16795607040
total_system_memory_human:15.64G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:437000
allocator_rss_ratio:1.35
allocator_rss_bytes:74293248
rss_overhead_ratio:0.91
rss_overhead_bytes:-25493504
mem_fragmentation_ratio:1.24
mem_fragmentation_bytes:49529160
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:249366647
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1570610391
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

Stats

total_connections_received:2
total_commands_processed:3
instantaneous_ops_per_sec:0
total_net_input_bytes:249333157
total_net_output_bytes:16
instantaneous_input_kbps:120600.10
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:1
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:3c767078d46a203529e05eabbce964153e41b7eb
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:0.411274
used_cpu_user:0.518047
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

Commandstats

cmdstat_ai.scriptset:calls=1,usec=1251631,usec_per_call=1251631.00
cmdstat_exists:calls=1,usec=2,usec_per_call=2.00
cmdstat_ping:calls=1,usec=2,usec_per_call=2.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=1,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=4 addr=127.0.0.1:46538 fd=13 name= age=5 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists
id=5 addr=[::1]:39200 fd=14 name= age=5 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=249332795 obl=0 oll=0 omem=0 events=r cmd=ai.modelset

------ CURRENT CLIENT INFO ------
id=5 addr=[::1]:39200 fd=14 name= age=5 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=249332795 obl=0 oll=0 omem=0 events=r cmd=ai.modelset
argv[0]: 'AI.MODELSET'
argv[1]: 'model'
argv[2]: 'TF'
argv[3]: 'CPU'
argv[4]: 'INPUTS'
argv[5]: 'input_1'
argv[6]: 'input_image_shape'
argv[7]: 'OUTPUTS'
argv[8]: 'concat_11'
argv[9]: 'concat_12'
argv[10]: 'concat_13'
argv[11]: '
[
input_1�
Placeholder*

dtype��0�6
shape�-:+�
�����������
�����������
������������
��
conv2d_1/kernel�Const��
�[��>�ҷ��ג���7��Z�=��׾�������=���>��q���T߾��>eO>�B��v�?h�L�ϼ
�=�&��#�����?d�����9>ڒ4��H�w�2>B���@
���ƾ'�2��v9����>����^ܲ<��a�L����
����>�9��>�$�=d�T��;ޖ����E>�>�����=�������<�P��h���L>o>�ɔ��x�by
�ڞ����.<�l�=-�����->��?���<X����=�[��D�
?0�V������o�>�$�>���=�^}=�;Q�|�;i�p�W�����ؽ�8�����>6��>ovм�#�=�/�67��V��=Ā�=�e�>�!���{�=M>�i���Pi�i�����7>X��Zk/<���>�_̿,I��n�

�ܥ�ݿ�>���>��W��ܭ����=��M>���=�N">%�D��߭>���=�r�>����R��>V����LU?��|=�"���p�?��=�m<�fh�����>�/���nݽ����T�t>L��=��F>�G@�|X�n������=L�¾�>�q
��nZ>��g>����P7>#
r���U=nT��o�h=P����A=<�/�>��Ǿ�[O���<b?�?��ս;����=�:�g%?2Ք?'�=;C3���b>���>���?�4��|�R=O�%?&��>2�^��R�=Nl�=��v<��>���>0���d̹���8�m�w?��a>~����_T=u6*�d�ý��>�cu���-?R���`դ����<�!c��T/�����q���AT��@�<٦
�̥(>�[ؽ8{⼳�X?�'

------ REGISTERS ------
80544:M 09 Oct 2019 08:40:50.859 #
RAX:0000560961ffe4e0 RBX:00007ffd733f0650
RCX:0000000000000000 RDX:00005609619e7010
RDI:00007ffd733f0630 RSI:00007ffd733f0540
RBP:00007ffd733f0710 RSP:00007ffd733f0600
R8 :0000560961f5b8b0 R9 :00005609620710f8
R10:0000000000000027 R11:0000000000000001
R12:00007ffd733f1108 R13:00005609619e7ec0
R14:00007ff15cd37390 R15:0000560962075230
RIP:00007ff15bb4fa59 EFL:0000000000010202
CSGSFS:002b000000000033
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060f) -> 0000000000000000
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060e) -> 00007ffd733f0730
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060d) -> 0000000000000000
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060c) -> 00007ff1a89b53e0
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060b) -> 00007ff1a7ddf8f8
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f060a) -> 0000000000000348
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0609) -> 00007ff1a87b5c09
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0608) -> 0000000000000001
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0607) -> 00007ff1a87b5af9
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0606) -> 0000002b00000007
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0605) -> 0000560962075230
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0604) -> 00007ff165df7b90
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0603) -> 0000000000000007
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0602) -> 00007ffd733f0760
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0601) -> 00007ff165dfac90
80544:M 09 Oct 2019 08:40:50.860 # (00007ffd733f0600) -> 0000000000000000

------ FAST MEMORY TEST ------
80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #0 terminated
80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #1 terminated
80544:M 09 Oct 2019 08:40:50.860 # Bio thread for job type #2 terminated
*** Preparing to test memory region 56095fc6e000 (2252800 bytes)
*** Preparing to test memory region 5609619e7000 (7188480 bytes)
*** Preparing to test memory region 7ff15cdcb000 (69632 bytes)
*** Preparing to test memory region 7ff165d7f000 (491520 bytes)
*** Preparing to test memory region 7ff166000000 (408944640 bytes)
*** Preparing to test memory region 7ff17e66d000 (335544320 bytes)
*** Preparing to test memory region 7ff192e34000 (16384 bytes)
*** Preparing to test memory region 7ff193082000 (8192 bytes)
*** Preparing to test memory region 7ff19ef6c000 (278528 bytes)
*** Preparing to test memory region 7ff19f6be000 (3670016 bytes)
*** Preparing to test memory region 7ff1a0936000 (4096 bytes)
*** Preparing to test memory region 7ff1a1a4c000 (12288 bytes)
*** Preparing to test memory region 7ff1a207c000 (3145728 bytes)
*** Preparing to test memory region 7ff1a2580000 (6815744 bytes)
*** Preparing to test memory region 7ff1a2dd1000 (8388608 bytes)
*** Preparing to test memory region 7ff1a35d2000 (8388608 bytes)
*** Preparing to test memory region 7ff1a3dd3000 (8388608 bytes)
*** Preparing to test memory region 7ff1a4dbc000 (139264 bytes)
*** Preparing to test memory region 7ff1a4ddf000 (8388608 bytes)
*** Preparing to test memory region 7ff1a57fe000 (8388608 bytes)
*** Preparing to test memory region 7ff1a5fff000 (8388608 bytes)
*** Preparing to test memory region 7ff1a6800000 (8388608 bytes)
*** Preparing to test memory region 7ff1a7000000 (8388608 bytes)
*** Preparing to test memory region 7ff1a7ddc000 (16384 bytes)
*** Preparing to test memory region 7ff1a7ffb000 (16384 bytes)
*** Preparing to test memory region 7ff1a89ae000 (32768 bytes)
*** Preparing to test memory region 7ff1a89d2000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /home/onstak/EdgeRealtimeVideoAnalytics/RedisAI/install-cpu/backends/redisai_tensorflow/lib/libtensorflow_framework.so.1 (base 0x7ff15b42a000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/antirez/redis/issues

Suspect RAM error? Use redis-server --test-memory to verify it.

Illegal instruction
I think this error generated because I have no avx and avx2 that's why Tensorflow not build correctly when I build redisai with bash get_deps.sh cpu after this error I installed tensorflow manully at redisgear successfully
i.e
/opt/redislabs/lib/modules/python3

import tensorflow as tf
print(tf.version)
1.12.0
but I have same Error 80544:M 09 Oct 2019 08:40:50.269 # Backend TF not loaded, will try loading default backend and redis has broken.
so how can I install Tensorflow 1.14.0 without AVX and AVX to load Tensorflow successfully

Machine : ubuntu 18.04
Python :3.7
AVX , AVX2 ,MKL and FMA : NO

Thanks

[MeirShpilraien]

@Akhtar303 Can you send me you redisai_tensorflow.so file? Maybe I will be able to understand from the stack trace where it crashed?

[MeirShpilraien]

Also how did you compiled redisai?

[Akhtar303]

Also how did you compiled redisai?
@MeirShpilraien
I follow these steps
bash get_deps.sh cpu
mkdir build
cd build
cmake ..
make && make install
cd ..
Thanks

[Akhtar303]

@Akhtar303 Can you send me you redisai_tensorflow.so file? Maybe I will be able to understand from the stack trace where it crashed?
@MeirShpilraien Thanks
Here is Link
https://drive.google.com/file/d/1Q4OAlWkTrg_vObCS482Ohz9f9R_2EmRX/view
Thanks

[Akhtar303]

@MeirShpilraien have you find any Cause for this problem.
Thanks @MeirShpilraien