Bug 1083961 - Every salt command produces multiple tracebacks
Every salt command produces multiple tracebacks
Status: RESOLVED WONTFIX
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Other
Current
Other Other
: P5 - None : Critical (vote)
: ---
Assigned To: E-Mail List
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2018-03-05 14:16 UTC by Nathan Cutler
Modified: 2018-05-01 08:26 UTC (History)
4 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nathan Cutler 2018-03-05 14:16:28 UTC
This is on openSUSE Leap 15.0, but I have anecdotal evidence that it affects Tumbleweed as well.

Python 2 is not installed.

# rpm -qi salt | egrep '^Version|^Release'
Version     : 2017.7.2
Release     : lp150.5.2

Every salt command produces multiple spam tracebacks like so:

# salt '*' test.ping                   
[WARNING ] /usr/lib/python3.6/site-packages/salt/utils/async.py:56: DeprecationWarning: zmq.eventloop.ioloop is deprecated in pyzmq 17. pyzmq now works with default tornado and asyncio eventloops.
  self.io_loop = LOOP_CLASS()

Exception ignored in: <bound method AsyncZeroMQReqChannel.__del__ of <salt.transport.zeromq.AsyncZeroMQReqChannel object at 0x7f39e8b02be0>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 160, in __del__
    self.message_client.destroy()
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 864, in destroy
    message_client.destroy()
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 919, in destroy
    self.stream.io_loop.remove_handler(self.stream.socket)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 734, in remove_handler
    fd, obj = self.split_fd(fd)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 656, in split_fd
    return fd.fileno(), fd
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/socket.py", line 157, in fileno
    return self.FD
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 45, in __getattr__
    return self._get_attr_opt(upper_key, opt)
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
    return self.get(opt)
  File "zmq/backend/cython/socket.pyx", line 477, in zmq.backend.cython.socket.Socket.get
  File "zmq/backend/cython/socket.pyx", line 137, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket
Exception ignored in: <bound method AsyncReqMessageClientPool.__del__ of <salt.transport.zeromq.AsyncReqMessageClientPool object at 0x7f39e8b1d470>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 860, in __del__
    self.destroy()
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 864, in destroy
    message_client.destroy()
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 919, in destroy
    self.stream.io_loop.remove_handler(self.stream.socket)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 734, in remove_handler
    fd, obj = self.split_fd(fd)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 656, in split_fd
    return fd.fileno(), fd
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/socket.py", line 157, in fileno
    return self.FD
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 45, in __getattr__
    return self._get_attr_opt(upper_key, opt)
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
    return self.get(opt)
  File "zmq/backend/cython/socket.pyx", line 477, in zmq.backend.cython.socket.Socket.get
  File "zmq/backend/cython/socket.pyx", line 137, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket
target167114245022.teuthology:
    True
Exception ignored in: <bound method AsyncReqMessageClient.__del__ of <salt.transport.zeromq.AsyncReqMessageClient object at 0x7f39e8537c18>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 928, in __del__
    self.destroy()
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 919, in destroy
    self.stream.io_loop.remove_handler(self.stream.socket)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 734, in remove_handler
    fd, obj = self.split_fd(fd)
  File "/usr/lib64/python3.6/site-packages/tornado/ioloop.py", line 656, in split_fd
    return fd.fileno(), fd
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/socket.py", line 157, in fileno
    return self.FD
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 45, in __getattr__
    return self._get_attr_opt(upper_key, opt)
  File "/usr/lib64/python3.6/site-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
    return self.get(opt)
  File "zmq/backend/cython/socket.pyx", line 477, in zmq.backend.cython.socket.Socket.get
  File "zmq/backend/cython/socket.pyx", line 137, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The same command run via salt-call --local does not have the problem:

# systemctl stop salt-minion.service
# systemctl stop salt-master.service
# salt-call --local '*' test.ping
'*' is not available.

The tracebacks can be silenced by running with "2>/dev/null". As far as I can tell they are just cosmetic, with no effect on functionality. Still, I'm marking the bug as critical because the effect on interactive usability is devastating.
Comment 1 Bo Maryniuk 2018-04-30 08:45:13 UTC
I'd recommend to upgrade to 2018.3 for this topic. The 2017.7 does not contain fix for this (yet).
Comment 2 Jochen Breuer 2018-04-30 09:04:35 UTC
According to Bo, we would need to backport this pull request to 2017.7.2: https://github.com/saltstack/salt/pull/46002
Comment 3 Klaus Kämpf 2018-04-30 10:11:56 UTC
(In reply to Jochen Breuer from comment #2)
> According to Bo, we would need to backport this pull request to 2017.7.2:
> https://github.com/saltstack/salt/pull/46002

No, we will not work on 2017.7.2. This version has never been maintained.

The question is, why does Leap 15 have 2017.7.2 while it should have 2018.3.0 ??
Comment 4 Bo Maryniuk 2018-04-30 11:28:04 UTC
(In reply to Klaus Kämpf from comment #3)
> The question is, why does Leap 15 have 2017.7.2 while it should have
> 2018.3.0 ??

Leap 15 has 2018.3.0:
https://build.opensuse.org/package/show/openSUSE:Leap:15.0/salt

Nathan, where did you go that version from?
Comment 5 Klaus Kämpf 2018-04-30 11:30:52 UTC
(In reply to Bo Maryniuk from comment #4)
> 
> Leap 15 has 2018.3.0:
> https://build.opensuse.org/package/show/openSUSE:Leap:15.0/salt

In this case, can't the bug be simply closed as fixed ?
Comment 6 Bo Maryniuk 2018-04-30 12:35:21 UTC
(In reply to Klaus Kämpf from comment #5)
> (In reply to Bo Maryniuk from comment #4)
> > 
> > Leap 15 has 2018.3.0:
> > https://build.opensuse.org/package/show/openSUSE:Leap:15.0/salt
> 
> In this case, can't the bug be simply closed as fixed ?

I think so, unless there are other version-specific issues. But that would I prefer as another bug.

Actually I explicitly fixed exactly this issue for ZMQ 17, which is different than predecessors (our original was 14). So in 2018.3 this is fixed, in 2017.7 is not, as we not even support this version.

I would just recommend upgrade Salt to the latest version and that's it.
Comment 7 Bo Maryniuk 2018-04-30 12:36:48 UTC
Resolution: please upgrade to the supported version 2018.3
Comment 8 Nathan Cutler 2018-05-01 08:26:35 UTC
> Nathan, where did you go that version from?

When I opened this bug, the version of Salt in SLE-15 was 2017.7.2. Now that we are on Salt 2018.03 we don't have this problem anymore. As long as no shipping product gets updated to Salt 2017 I think it's fine to close the bug.

Thanks!