Bug 1150807 - NFS-client is broken, nfs.service does not exist.
NFS-client is broken, nfs.service does not exist.
Status: RESOLVED FIXED
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Network
Current
x86-64 Other
: P5 - None : Critical (vote)
: ---
Assigned To: Neil Brown
E-mail List
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2019-09-15 18:44 UTC by e j
Modified: 2020-03-27 04:35 UTC (History)
8 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Attachments
command showmount (111 bytes, text/plain)
2019-09-17 06:05 UTC, Sebastian Kuhne
Details
command grep (224 bytes, text/plain)
2019-09-17 06:06 UTC, Sebastian Kuhne
Details
command mount (246 bytes, text/plain)
2019-09-17 06:06 UTC, Sebastian Kuhne
Details

Note You need to log in before you can comment on or make changes to this bug.
Description e j 2019-09-15 18:44:34 UTC
After update to Tumbleweed 20190909:
 
Existing NFS share fails to mount at boot as normal. Running yast nfs client returns following message on initialization:

These packages need to be installed: nfsidmap

allow install results in the following error message:

Installing required packages failed. If you continue without installing required packages, Yast may not work properly.

If process is continued Yast fails to start NFS.

Additionally nfs.service seems to not exist:

# systemctl start nfs.service 
Failed to start nfs.service: Unit nfs.service not found.

# systemctl status nfs.service 
● nfs.service
   Loaded: not-found (Reason: Unit nfs.service not found.)
   Active: active (exited) since Sat 2019-09-14 22:28:27 PDT; 12h ago
 Main PID: 15926 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs.service

 # systemctl status nfs-idmapd.service 
● nfs-idmapd.service - NFSv4 ID-name mapping service
   Loaded: loaded (/usr/lib/systemd/system/nfs-idmapd.service; static; vendor preset: disabled)
   Active: active (running) since Sun 2019-09-15 11:29:28 PDT; 4s ago
  Process: 10919 ExecStart=/usr/sbin/rpc.idmapd (code=exited, status=0/SUCCESS)
 Main PID: 10920 (rpc.idmapd)
    Tasks: 1 (limit: 4915)
   Memory: 396.0K
   CGroup: /system.slice/nfs-idmapd.service
           └─10920 /usr/sbin/rpc.idmapd

Sep 15 11:29:28 linux-ka5p systemd[1]: Stopping NFSv4 ID-name mapping service...
Sep 15 11:29:28 linux-ka5p systemd[1]: nfs-idmapd.service: Main process exited, code=killed, status=15/TERM
Sep 15 11:29:28 linux-ka5p systemd[1]: nfs-idmapd.service: Succeeded.
Sep 15 11:29:28 linux-ka5p systemd[1]: Stopped NFSv4 ID-name mapping service.
Sep 15 11:29:28 linux-ka5p systemd[1]: Starting NFSv4 ID-name mapping service...
Sep 15 11:29:28 linux-ka5p rpc.idmapd[10919]: rpc.idmapd: config error at /etc/nfs.conf:12: error loading included config
Sep 15 11:29:28 linux-ka5p rpc.idmapd[10919]: config error at /etc/nfs.conf:12: error loading included config
Sep 15 11:29:28 linux-ka5p rpc.idmapd[10920]: Setting log level to 0
Sep 15 11:29:28 linux-ka5p systemd[1]: Started NFSv4 ID-name mapping service.

Operating System: openSUSE Tumbleweed 20190909
KDE Plasma Version: 5.16.5
KDE Frameworks Version: 5.61.0
Qt Version: 5.13.1
Kernel Version: 5.2.11-1-default
OS Type: 64-bit
Processors: 2 × Intel® Celeron® 2955U @ 1.40GHz
Memory: 7.7 GiB of RAM

Identical behavior observed on separate machine running kernel 4.20.0-1-default
Comment 1 Sebastian Kuhne 2019-09-16 04:31:46 UTC
Probably the same or a similar issue is beeing discussed here:

https://lists.opensuse.org/opensuse-support/2019-09/msg00070.html
https://lists.opensuse.org/opensuse-factory/2019-09/msg00132.html
Comment 2 Sebastian Kuhne 2019-09-16 04:34:47 UTC
My setup:

- one NFS server running TW 20190909
- three NFS clients running Leap 15.1.

All updates in place.

The NFS setup was running stable for years.
Since TW 20190909 installed on the server, access from all clients to the server doesn't work anymore.
Comment 3 Neil Brown 2019-09-16 05:52:37 UTC
yast shouldn't encode a dependency on nfsidmap - that is that job of that package management - not yast.

libnfsidmap has been merged into nfs-utils, so there is no separate package any more.

nfs.service is a legacy name.  nfs-client.service is the name that upstream uses and has been available for quite some time.
We recently removed the legacy names.

nfs-client runs client-side services.
nfs-server runs server-side services.

However it seems that yast still uses "nfs".  I thought I had asked them to change that.

The error message
 rpc.idmapd: config error at /etc/nfs.conf:12: error loading included config

does not signify a fatal error.  You can silence it by creating /etc/nfs.conf.local as an empty file.  I'll get that sorted out in the next update.

If you 
  systemctl start nfs-client
and the
  mount -a

does the NFS get mounted properly?

yast2-maintaineres: can the dependecy on nfsidmap be removed, and the service names be changed to nfs-client and nfs-server?

Also the old nfs-client script mounted NFS filesystems, while the new nfs-client service doesn't do that.  So after adding things to /etc/fstab, yast would need to "mount -a -t nfs" or similar. (this was discussed in Bug 1006815 )
Comment 4 Neil Brown 2019-09-16 06:16:21 UTC
Thinking about this a bit more...
The above comments don't address why the filesystems weren't mounted on reboot.  yast doesn't do anything at that point.

Can you please try
  systemctl enable nfs-client
and then reboot - and report if the filesystems were mounted successful.
I might need to get the package update process to check if 'nfs' as enabled, and if so: enable nfs-client.
Comment 5 Neil Brown 2019-09-16 06:19:23 UTC
Sorry, nfs-client is a target, not a service.
So you need
  systemctl enable nfs-client.target
Comment 6 Sebastian Kuhne 2019-09-16 06:29:40 UTC
(In reply to Neil Brown from comment #5)
> Sorry, nfs-client is a target, not a service.
> So you need
>   systemctl enable nfs-client.target

Done on one leap client, but the problem still persists.
Comment 7 e j 2019-09-16 07:02:54 UTC
On tumbleweed systeme I did: 
#systemctl enable nfs-client.target 
followed by reboot.  NFS still failed to mount.  Failed to start with [FAILED] NIS/YP Clients to NIS Doamin Binder. See systemctl status ypbind.service for details.

# systemctl status ypbind.service 
● ypbind.service - NIS/YP Clients to NIS Domain Binder
   Loaded: loaded (/usr/lib/systemd/system/ypbind.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2019-09-15 23:45:43 PDT; 9min ago
     Docs: man:ypbind(8)
  Process: 1741 ExecStartPre=/usr/lib/ypbind/ypbind-systemd-pre (code=exited, status=1/FAILURE)
  Process: 1750 ExecStopPost=/bin/sh -c /bin/rm -f /var/yp/binding/* /var/run/ypbind.pid (code=exited, status=0/SUCCESS)

Sep 15 23:45:43 linux-ka5p systemd[1]: Starting NIS/YP Clients to NIS Domain Binder...
Sep 15 23:45:43 linux-ka5p ypbind-systemd-pre[1741]: \nError: NIS domain not specified.\n
Sep 15 23:45:43 linux-ka5p systemd[1]: ypbind.service: Control process exited, code=exited, status=1/FAILURE
Sep 15 23:45:43 linux-ka5p systemd[1]: ypbind.service: Failed with result 'exit-code'.
Sep 15 23:45:43 linux-ka5p systemd[1]: Failed to start NIS/YP Clients to NIS Domain Binder.

Also I'm wondering if the server side is also broken because in below only the TW server was updated and the Leap clients should be unchanged:

Comment # 2 on bug 1150807 from Sebastian Kuhne
My setup:

- one NFS server running TW 20190909
- three NFS clients running Leap 15.1.

All updates in place.

The NFS setup was running stable for years.
Since TW 20190909 installed on the server, access from all clients to the
server doesn't work anymore.

In my case both client and server are TW and both were updated but the server runs kernel 4.20.0-1-default
Comment 8 Sebastian Kuhne 2019-09-16 07:23:37 UTC
> Also I'm wondering if the server side is also broken because in below only
> the TW server was updated and the Leap clients should be unchanged:
> 
> Comment # 2 on bug 1150807 from Sebastian Kuhne
> My setup:
> 
> - one NFS server running TW 20190909
> - three NFS clients running Leap 15.1.
> 
> All updates in place.
> 
> The NFS setup was running stable for years.
> Since TW 20190909 installed on the server, access from all clients to the
> server doesn't work anymore.
> 
> In my case both client and server are TW and both were updated but the
> server runs kernel 4.20.0-1-default

This is my assumption, too. I think it is a SERVER issue and not a client issue. The three clients were unchanged and not updated - only the server got the TW 20190909 update.
Comment 9 Neil Brown 2019-09-16 07:29:59 UTC
Does
  systemctl enable nfs-server
  systemctl strat nfs-server

on the server help?

The ypbind message suggest that /etc/defaultdomain doesn't exist, or is empty.
I cannot see how that would be related to nfs packages.
Comment 10 e j 2019-09-16 14:21:55 UTC
Below is the result on the TW server which indicates an error "config error at /etc/nfs.conf:12: error loading included config."

Valhalla:/etc # systemctl start nfs-server  
Valhalla:/etc # systemctl status nfs-server 
● nfs-server.service - NFS server and services 
  Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) 
 Drop-In: /usr/lib/systemd/system/nfs-server.service.d 
          └─options.conf 
          /run/systemd/generator/nfs-server.service.d 
          └─order-with-mounts.conf 
  Active: active (exited) since Mon 2019-09-16 06:02:38 PDT; 1min 25s ago 
Main PID: 4684 (code=exited, status=0/SUCCESS) 
   Tasks: 0 (limit: 4915) 
  Memory: 0B 
  CGroup: /system.slice/nfs-server.service 

Sep 16 06:02:38 Valhalla.localdomain systemd[1]: Starting NFS server and services... 
Sep 16 06:02:38 Valhalla.localdomain exportfs[4683]: exportfs: config error at /etc/nfs.conf:12: error loading included config 
Sep 16 06:02:38 Valhalla.localdomain rpc.nfsd[4684]: : config error at /etc/nfs.conf:12: error loading included config 
Sep 16 06:02:38 Valhalla.localdomain systemd[1]: Started NFS server and services. 
Valhalla:/etc # ls /etc/nfs 
nfs.conf       nfsmount.conf   
Valhalla:/etc # ls /etc/nfs.conf 
/etc/nfs.conf 
Valhalla:/etc # cat /etc/nfs.conf 
# 
# This is a general configuration for the 
# NFS daemons and tools 
# DO NOT MAKE CHANGES TO THIS FILE as they will 
# be lost on the next software update.  Make changes 
# to /etc/sysconfig/nfs or /etc/nfs.conf.local instead. 
# /etc/nfs.conf.local can include multiple sections, just 
# like this file. 

[environment] 
include = /etc/sysconfig/nfs 
include = /etc/nfs.conf.local 
[general] 
pipefs-directory=$RPC_PIPEFS_DIR 
# 
#[exportfs] 
# debug=0 
# 
#[gssd] 
# use-memcache=0 
# use-machine-creds=1 
avoid-dns=$NFS_GSSD_AVOID_DNS 
# limit-to-legacy-enctypes=0 
# context-timeout=0 
# rpc-timeout=5 
# keytab-file=/etc/krb5.keytab 
# cred-cache-directory= 
# preferred-realm= 
# 
[lockd] 
port=$LOCKD_TCPPORT 
udp-port=$LOCKD_UDPPORT 
# 
[mountd] 
# debug=0 
# manage_gids=n 
# descriptors=0 
port= $MOUNTD_PORT 
# threads=1 
# reverse-lookup=n 
# state-directory-path=/var/lib/nfs 
# ha-callout= 
# 
#[nfsdcltrack] 
# debug=0 
# storagedir=/var/lib/nfs/nfsdcltrack 
# 
[nfsd] 
# debug=0 
threads= $USE_KERNEL_NFSD_NUMBER 
# host= 
# port=0 
# grace-time=90 
lease-time=$NFSV4LEASETIME 
# udp=y 
# tcp=y 
# vers2=n 
vers3=$NFS3_SERVER_SUPPORT 
vers4=$NFS4_SUPPORT 
# vers4.0=y 
# vers4.1=y 
# vers4.2=y 
# rdma=n 
# 
[statd] 
# debug=0 
port=$STATD_PORT 
# outgoing-port=0 
name=$STATD_HOSTNAME 
# state-directory-path=/var/lib/nfs/statd 
# ha-callout= 
# 
#[sm-notify] 
# debug=0 
# retry-time=900 
# outgoing-port= 
# outgoing-addr= 
# 
#[svcgssd] 
# principal= 
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc # showmount -e localhost 
Export list for localhost: 
/mnt/RAID1 * 
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc #  
Valhalla:/etc # systemctl restart nfs-server  
Valhalla:/etc # systemctl status nfs-server  
● nfs-server.service - NFS server and services 
  Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) 
 Drop-In: /usr/lib/systemd/system/nfs-server.service.d 
          └─options.conf 
          /run/systemd/generator/nfs-server.service.d 
          └─order-with-mounts.conf 
  Active: active (exited) since Mon 2019-09-16 06:08:31 PDT; 4s ago 
 Process: 5458 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) 
 Process: 5459 ExecStart=/usr/sbin/rpc.nfsd $NFSD_OPTIONS (code=exited, status=0/SUCCESS) 
Main PID: 5459 (code=exited, status=0/SUCCESS) 

Sep 16 06:08:31 Valhalla.localdomain systemd[1]: Starting NFS server and services... 
Sep 16 06:08:31 Valhalla.localdomain exportfs[5458]: exportfs: config error at /etc/nfs.conf:12: error loading included config 
Sep 16 06:08:31 Valhalla.localdomain rpc.nfsd[5459]: : config error at /etc/nfs.conf:12: error loading included config 
Sep 16 06:08:31 Valhalla.localdomain systemd[1]: Started NFS server and services. 
Valhalla:/etc # showmount -e localhost       
Export list for localhost: 
/mnt/RAID1 * 
Valhalla:/etc #
Comment 11 Sebastian Kuhne 2019-09-16 17:56:39 UTC
(In reply to Neil Brown from comment #9)
> Does
>   systemctl enable nfs-server
>   systemctl strat nfs-server
> 
> on the server help?
> 
> The ypbind message suggest that /etc/defaultdomain doesn't exist, or is
> empty.
> I cannot see how that would be related to nfs packages.

Unfortunately, this did not help. Actually as expected since the server was working. So, no reason to enable and to start a running system.

There must be another root cause.
Comment 12 Neil Brown 2019-09-16 23:39:32 UTC
Thanks for testing - sorry we haven't found the problem yet.

The "config error at /etc/nfs.conf:12: error loading included config" message are just noise - they aren't causing failure.  They will go away in the next update but are not related to the problem.

On a client, please run:

showmount -e $SERVER
grep nfs /etc/fstab
mount -v -a -t nfs,nfs4

and report the output - hopefully that will show me what is going wrong.
Comment 13 Swamp Workflow Management 2019-09-17 02:00:10 UTC
This is an autogenerated message for OBS integration:
This bug (1150807) was mentioned in
https://build.opensuse.org/request/show/731364 Factory / nfs-utils
Comment 14 e j 2019-09-17 02:31:57 UTC
(In reply to Neil Brown from comment #12)
> Thanks for testing - sorry we haven't found the problem yet.
> 
> The "config error at /etc/nfs.conf:12: error loading included config"
> message are just noise - they aren't causing failure.  They will go away in
> the next update but are not related to the problem.
> 
> On a client, please run:
> 
> showmount -e $SERVER
> grep nfs /etc/fstab
> mount -v -a -t nfs,nfs4
> 
> and report the output - hopefully that will show me what is going wrong.

In reply to Neil Brown from comment #12)
> Thanks for testing - sorry we haven't found the problem yet.
> 
> The "config error at /etc/nfs.conf:12: error loading included config"
> message are just noise - they aren't causing failure.  They will go away in
> the next update but are not related to the problem.
> 
> On a client, please run:
> 
> showmount -e $SERVER
> grep nfs /etc/fstab
> mount -v -a -t nfs,nfs4
> 
> and report the output - hopefully that will show me what is going wrong.

I think this is partly a firewall. If I try to mount using "mount -v -a -t nfs,nfs4," I get no path to server (see below) But if I disable the server firewall i can successfully mount the nfs. Also if I reboot the client with the server firewall disabled it boots normally and the nfs is mounted. I noticed nfs and nfs3 services in the "public" zone rather than "internal" which is where I think they should be so added those to "internal" but I still cannot successfully mount the nfs with the server firewall enabled.


Here is the out put from:

showmount -e $SERVER
grep nfs /etc/fstab
mount -v -a -t nfs,nfs4

# showmount -e $SERVER
clnt_create: RPC: Timed out (same result with firewall on or off)
linux-ka5p:/home/ej # grep nfs /etc/fstab
192.168.10.22:/mnt/RAID1                   /mnt/RAID1              nfs    rw                            0  0
linux-ka5p:/home/ej # mount -v -a -t nfs,nfs4  (server firewall is ON)
/                        : ignored
swap                     : ignored
/var                     : ignored
/usr/local               : ignored
/tmp                     : ignored
/srv                     : ignored
/root                    : ignored
/opt                     : ignored
/home                    : ignored
/boot/grub2/x86_64-efi   : ignored
/boot/grub2/i386-pc      : ignored
/boot/efi                : ignored
mount.nfs: timeout set for Mon Sep 16 17:36:46 2019
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query failed: RPC: Remote system error - No route to host
mount.nfs: No route to host

Seeing no route to host I thought maybe to check firewall on server and so I tried to stop firewall and recheck mount nfs on client and:

linux-ka5p:/home/ej # mount -v -a -t nfs,nfs4 (server firewall is OFF)
/                        : ignored
swap                     : ignored
/var                     : ignored
/usr/local               : ignored
/tmp                     : ignored
/srv                     : ignored
/root                    : ignored
/opt                     : ignored
/home                    : ignored
/boot/grub2/x86_64-efi   : ignored
/boot/grub2/i386-pc      : ignored
/boot/efi                : ignored
mount.nfs: timeout set for Mon Sep 16 17:41:57 2019
mount.nfs: trying text-based options 'vers=4.2,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,minorversion=1,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'vers=4,addr=192.168.10.22,clientaddr=192.168.10.7'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.10.22'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.10.22 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.10.22 prog 100005 vers 3 prot UDP port 20048
/mnt/RAID1               : successfully mounted
Comment 15 Neil Brown 2019-09-17 03:01:05 UTC
Apparently SuSEfirewall2 has been replaced by firewalld.
I know nothing about this, but Matthias made a related change to nfs-utils.
Maybe he can help.

Matthias???
Comment 16 Sebastian Kuhne 2019-09-17 06:03:17 UTC
I am attaching the three commands as text files now.
Comment 17 Sebastian Kuhne 2019-09-17 06:05:33 UTC
Created attachment 818510 [details]
command showmount
Comment 18 Sebastian Kuhne 2019-09-17 06:06:01 UTC
Created attachment 818511 [details]
command grep
Comment 19 Sebastian Kuhne 2019-09-17 06:06:38 UTC
Created attachment 818513 [details]
command mount
Comment 20 Sebastian Kuhne 2019-09-17 06:14:44 UTC
In my opinion, showmount and fstab look normal but mount did not behave as expected.

In terms of firewalld - on both, the server and the client, I have disabled firewalld. I did that since I know the firewall may cause some issues with nfs - and consequently the firewall was my first candidate to exclude. So, actually the firewall should not be the root cause.

Let me emphasize again, on the three client's side there was no update, no change of fstab, avahi, etc. The only change was the update on the server to TW 20190909. Besides this TW update, no other changes on the server for example in /etc/export.
Comment 21 Matthias Gerstner 2019-09-17 08:10:54 UTC
(In reply to Neil Brown from comment #15)
> Apparently SuSEfirewall2 has been replaced by firewalld.
> I know nothing about this, but Matthias made a related change to nfs-utils.
> Maybe he can help.

I helped with the migration back then but I'm no active maintainer of firewalld. I'm adding rfrohl from the security who is maintainer of firewalld these days.

There hasn't been an update to firewalld in Tumbleweed for months so I don't see why the firewalld should be directly to blame here. Apart from that a simple

```
firewall-cmd --add-service nfs
```

should work in a standard setup. At least when using NFSv4 where only TCP port 2049 is needed any more and no dynamically allocated rpcbind ports.

The SUSE security guide [1] offers an introduction to firewalld and also handles the topic of NFSv3 vs. NFSv4 regarding firewalld handling.

[1]: https://www.suse.com/documentation/sles-15/book_security/data/sec_security_firewall_firewalld.html
Comment 22 e j 2019-09-18 18:10:38 UTC
I have updated both my server and client to TW 20190916. 

After the update on the server I removed and re-added the NFS directory using the yast2 NFS server module. At first NFS was not mountable on the client. Out of frustration I then booted a second client (TW 20190902) that was running Win10 in dual boot and didn't receive the 20190909 update. NFS mounted at boot as usual! I then tried to mount NFS manually on the TW 20190916 client which was successful. To confirm normal behavior the 20190916 client was then rebooted and now NFS mounted successfully at boot. I'm not sure why it didn't work initially but everything seems back to normal now. 

Can anyone else conform update to 20190916 restores NFS?
Comment 23 Sebastian Kuhne 2019-09-19 04:26:12 UTC
(In reply to e j from comment #22)
> I have updated both my server and client to TW 20190916. 
> 
My config:
Server: Updated to TW 20190916 yesterday
Client: Leap 15.1, no change at all
--> Still no access from client to server.

Seems that a TW installation on both, server and client may fix the issue now. But in case of a different setup the issue persists.
Comment 24 Neil Brown 2019-09-23 02:02:06 UTC
I think I've discovered the cause.
The message 
 config error at /etc/nfs.conf:12: error loading included config
should be non-fatal, but due to other bugs in the code, reporting an
error at this point in the code caused nfs.mountd to misbehave.

If you create /etc/nfs.conf.local and restart nfs-server, the problems
should go away.

Except for the fact that yast still used nfs.service and nfsserver.service...
Comment 25 Neil Brown 2019-09-23 02:08:42 UTC
I added  needinfo?(yast2-maintainers@suse.de)  earlier, but it seems to have been removed.

yast2-maintainers:  can the dependency on in nfsidmap package be removed (see comment #3, and can yast use nfs-client.target and nfs-server.service as the systemd units to manage NFS?
Also yast cannot depend on nfs-client.target to mount nfs filesystem - that needs to be done with "mount -a -t nfs,nfs4" or similar.
See bug 1006815.
Comment 26 Swamp Workflow Management 2019-09-23 02:50:09 UTC
This is an autogenerated message for OBS integration:
This bug (1150807) was mentioned in
https://build.opensuse.org/request/show/732555 Factory / nfs-utils
Comment 27 Sebastian Kuhne 2019-09-23 05:06:01 UTC
(In reply to Neil Brown from comment #24)
> I think I've discovered the cause.
> The message 
>  config error at /etc/nfs.conf:12: error loading included config
> should be non-fatal, but due to other bugs in the code, reporting an
> error at this point in the code caused nfs.mountd to misbehave.
> 
> If you create /etc/nfs.conf.local and restart nfs-server, the problems
> should go away.
> 
> Except for the fact that yast still used nfs.service and nfsserver.service...

Hi Neil,

I can confirm that after creating an empty /etc/nfs.conf.local at the server side the overall nfs service is back on all clients.

Many thanks for your efforts!
Comment 28 e j 2019-09-23 21:33:51 UTC
(In reply to Neil Brown from comment #24)
  
> If you create /etc/nfs.conf.local and restart nfs-server, the problems
> should go away.
> 
> Except for the fact that yast still used nfs.service and nfsserver.service...

Now that you mention it this makes sense. At some point while I was troubleshooting I created a blank /etc/nfs.conf.local. This would explain why my NFS started working while others still had a broken system.

Added thanks for all your work on this!
Comment 29 Neil Brown 2020-03-27 04:35:38 UTC
This seems to be all resolve now, so closing.