Databases, Systems & Networks

I want to find out how many times a word (say foo or an IP address) occurs in a text file using the grep command on Linux or Unix-like system?

You can use the grep command to search strings, words, text, and numbers for a given patterns. You can pass the -coption to grep command. It only shows the number of times that the pattern has been matched for each file.

Show the total number of times that the word foo appears in a file named bar.txt

The syntax is:
grep -c string filename
grep -c foo bar.txt
Sample outputs:

3

To count total number of occurrences of word in a file named /etc/passwd root using grep, run:
grep -c root /etc/passwd
To verify that run:
grep --color root /etc/passwd
Pass the -w option to grep to select only an entire word or phrase that matches the specified pattern:
grep -w root /etc/passwd
OR
grep -c -w root /etc/passwd
In this example only match a word being with root:
grep --color -w '^root' /etc/passwd
grep -c -w '^root' /etc/passwd
To show only the matching part of the lines.
grep -o 'root' /etc/passwd
grep -c -o 'root' /etc/passwd
Sample session:

Fig.01: Counting occurrence of words/strings using grep command

Copy files or directories from one location to an another host by rsync.

If you’d like to set rsync automatically by cron or others, it need to configure like follows because authentication is required without settings. For example, Copy files or directories under the [/root/work] on dlp.srv.world to [/home/backup] on www.srv.world.

[1] Configure on source host.

root@dlp:~# apt-get -y install rsync
root@dlp:~# vi /etc/rsync_exclude.lst
# specify files or directories you'd like to exclude to copy
test
test.txt

[2] Configure on destination host.

root@www:~# apt-get -y install rsync
root@www:~# vi /etc/default/rsync
# line 8: change
RSYNC_ENABLE=true
root@www:~# vi /etc/rsyncd.conf
# create new
# any name you like
[backup]
# destination directory to copy
path = /home/backup
# hosts you allow to access
hosts allow = 10.0.0.30
hosts deny = *
list = true
uid = root
gid = root
read only = false
root@www:~# mkdir /home/backup
root@www:~# systemctl start rsync

[3] It’s OK. Execute rsync on Source Host like follows.

root@dlp:~# rsync -avz --delete --exclude-from=/etc/rsync_exclude.lst /root/work/ www.srv.world::backup
# Add in cron if you'd like to run reguraly
root@dlp:~# crontab -e
# for example, run at 2:00 AM in a day
00 02 * * * rsync -avz --delete --exclude-from=/etc/rsync_exclude.lst /root/work/ www.srv.world::backup

If you read our previous article Easy Ubuntu Server Firewall, then you may have noted that on Ubuntu 16.04 the described method no longer works. This is due to systemd. In the article below we will walk through creating a persistent IPTables based firewall on Ubuntu 16.04 LTS. First we need to install some required software packages. As seen in the command below, install iptables-persistent. Next we will make netfilter-persistent run at boot. This is the most important step as it will ensure your rules are reloaded at boot time.

# Install IPTables Persistent Package
apt-get install -y iptables-persistent
# Add netfilter-persistent Startup
invoke-rc.d netfilter-persistent save
# Stop netfilter-persistent Service
service netfilter-persistent stop

Once the packages above are installed and the service is stopped, you will have a new directory at /etc/iptables/. This directory holds the IPTables filter rules that will be reloaded at boot time. These files are named rules.v4 and rules.v6 respectively. IPV4 rules are loaded into rules.v4 and IPV6 rules are loaded into rules.v6. For the purpose of this article we will focus on IPV4 rules. Next we will want to copy the rules below into our rules.v4 file. Of course the rules will need to be modified to fit your environment.

# Generated by iptables-save v1.3.3 on Wed Apr 9 10:51:08 2008
# Flush out any rules that are already in there
*filter
:INPUT ACCEPT [146:11332]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [104:9831]
 
# Allow internal loopback connections
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
 
# Allow pinging
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
 
# Allow any outbound data, and any inbound data related to a connection that is already in use
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
 
# =========BEGIN SERVER SPECIFIC PORT OPEN RULES=========
# Allow SCP/SSH Access from Green & Blue Subnet
-A INPUT -s 172.16.12.0/255.255.255.0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.10.12.0/255.255.255.0 -p tcp -m tcp --dport 22 -j ACCEPT
 
# Allow HTTP Access from Red Subnet/Internet
-A INPUT -p tcp -m state --state NEW,ESTABLISHED --dport 80 -j ACCEPT
 
# Allow HTTPS Access from Red Subnet/Internet
-A INPUT -p tcp -m state --state NEW,ESTABLISHED --dport 443 -j ACCEPT
 
# Allow MySQL Access from Red Subnet/Internet
-A INPUT -p tcp -m state --state NEW,ESTABLISHED --dport 3306 -j ACCEPT
 
# Allow FTP Access from Red Subnet/Internet
-A INPUT -p tcp -m state --state NEW,ESTABLISHED --dport 21 -j ACCEPT
-A INPUT -p tcp -m state --state NEW,ESTABLISHED --dport 58000:58010 -j ACCEPT
# =========END SERVER SPECIFIC PORT OPEN RULES=========
 
# Drop everything that hasn't been picked up by one of the rules above
-A INPUT -j DROP
-A FORWARD -j DROP
-A OUTPUT -j DROP
 
COMMIT
# Completed on Wed Apr 9 10:51:08 2008

Lastly, in order for our new rules to take affect, we simply need to start the netfilter-persistent service as seen below. That’s it, you now have a fully functional IPTables based firewall.

# Start netfilter-persistent Service
service netfilter-persistent start

# Check if IPTables were applied
iptables -L

there’s always apt-cache policy <package-name> (no sudo needed).

Not installed:

olivier@neews:/$ apt-cache policy gnuift
 gnuift:
   Installed: (none)
   Candidate: 0.1.14-11
   Version table:
      0.1.14-11 0
         500 http://archive.ubuntu.com/ubuntu/ oneiric/universe amd64 Packages

Installed:

olivier@neews:/$ apt-cache policy firefox
 firefox:
   Installed: 8.0+build1-0ubuntu0.11.10.3
   Candidate: 8.0+build1-0ubuntu0.11.10.3
   Version table:
  *** 8.0+build1-0ubuntu0.11.10.3 0
         500 http://archive.ubuntu.com/ubuntu/ oneiric-updates/main amd64 Packages
         500 http://archive.ubuntu.com/ubuntu/ oneiric-security/main amd64 Packages
         100 /var/lib/dpkg/status
      7.0.1+build1+nobinonly-0ubuntu2 0
         500 http://archive.ubuntu.com/ubuntu/ oneiric/main amd64 Packages

Or dpkg: dpkg -l | grep -E '^ii' | grep <package name>. When it’s not installed it won’t show output. When it is, it’ll show something like:

olivier@neews:~$ dpkg -l | grep -E '^ii' | grep firefox
 ii  firefox                                                     8.0+build1-0ubuntu0.11.10.3                            Safe and easy web browser from Mozilla
 ii  firefox-branding                                            8.0+build1-0ubuntu0.11.10.3                            Safe and easy web browser from Mozilla - transitional package
 ii  firefox-globalmenu                                          8.0+build1-0ubuntu0.11.10.3                            Unity appmenu integration for Firefox
 ii  firefox-gnome-support                                       8.0+build1-0ubuntu0.11.10.3                            Safe and easy web browser from Mozilla - GNOME support
 ii  firefox-locale-en                                           8.0+build1-0ubuntu0.11.10.3                            English language pack for Firefox

It's obviously a fuzzier search but handy if you're not sure which package you're looking for.
 
 For manually installed things...
 
 A bit harder but if they're on the current path, you could just run them. That's a bit of mission so I'd rather just run:
 
 oli@bert:/$ which chromium-browser
 /usr/bin/chromium-browser

And:

oli@bert:/$ which gnuift
# returns nothing

Which is better?

That depends on the sanity of user. There’s nothing to stop somebody installing something called chromium-browser that isn’t Chromium. They could even package it up incorrectly and install that. Neither method can be 100% certain.

But assuming the owner is sane – packages should be good enough for most people.

To clone a Debian installation, use the apt-clone utility. It’s available (as a separate package, not part of the default installation) in Debian since wheezy and in Ubuntu since 12.04. On the existing machine, run

apt-clone clone foo

This creates a file foo.apt-clone.tar.gz. Copy it to the destination machine, and run

apt-get install apt-clone
apt-clone restore foo.apt-clone.tar.gz

If you’re working with an old system where apt-clone isn’t available, or if you just want to replicate the list of installed packages but not any configuration file, here are the manual steps.

• On the source machine:

cat /etc/apt/sources.list /etc/apt/sources.list.d >sources.list
pkg --get-selections >selections.list
apt-mark auto >auto.list

• On the target machine:

cp sources.list /etc/apt/
apt-get update
/usr/lib/dpkg/methods/apt/update /var/lib/dpkg/
dpkg --set-selections <selections.list
apt-get dselect-upgrade
xargs apt-mark auto <auto.list

I believe that you’re affected by an incompatible change in dpkg that first made it into wheezy. See bug #703092 for background.

The short story is that dpkg --set-selections now only accepts package names that are present in the file /var/lib/dpkg/status or /var/lib/dpkg/available. If you only use APT to manage packages, like most people, then /var/lib/dpkg/available is not kept up-to-date.

After running apt-get update and before running dpkg --set-selections and apt-get -u dselect-upgrade, run the following command:

apt-cache dumpavail >/tmp/apt.avail
dpkg --merge-avail /tmp/apt.avail

From jessie onwards, you can simplify this to

apt-cache dumpavail | dpkg --merge-avail

Alternatively, run

/usr/lib/dpkg/methods/apt/update /var/lib/dpkg/

or even simpler

apt-get install dctrl-tools
sync-available

Another simple method that doesn’t require installing an additional package but will download the package lists again is

dselect update

See the dpkg FAQ for more information. (This is mentioned in the dpkg man page, but more in a way that would remind you of the issue if you were already aware, not in a way that explains how to solve the problem!)

Note that cloning a package installation with dpkg --set-selections doesn’t restore the automatic/manual mark in APT. See Restoring all data and dependencies from dpkg –set-selections ‘*’ for more details. You can save the marks on the source system with

apt-mark showauto >auto.list

and restore them on the target system with

xargs apt-mark auto <auto.list