Tuesday, January 31, 2017

From X10 to Z-Wave with Alexa and Google Home: The Sanyal Family DIY Smart Home Project





Introduction


I bought the revolutionary Amazon Echo with Alexa voice assistance just after its initial public release around mid-2015. The little Alexa RF Remote came in the box with the original Echo - these days Amazon sells the remote separately!

Ever since obtaining the Amazon Echo, I have been watching the popular interest in home automation grow. It has now become mainstream with non-geeks jumping in to buy packaged solutions. The sudden rush is somewhat humorous - I have been playing with X10 based home automation devices and systems for years - but the recent developments do make smart homes much easier to implement, and also address a major drawback of X10: security. However, X10 remains incredibly affordable - one can still get an entire entry-level X10 home automation kit for the price of a single Z-Wave garage door opener!

What does the Amazon Echo with Alexa have to do with home automation? It provides voice control! That is indeed a leap from fooling around with computer applications, or phone applications, and this voice control integration with the Amazon Echo is perhaps the greatest motivator for home automation resulting in the plethora of products and protocols competing in the non-hobbyist user market now.

The other enabler for the mass consumer interest in home automation is the smart phone and the apps that run on it. Being able to set your home thermostat to a balmy 72 degrees using your smart phone before you head home from work or taking a peek at a video stream of your living room when on vacation in Honolulu (unless you are fortunate enough to live in Honolulu) are genuinely useful technological conveniences, not just a way to bragging rights.

But Alexa, and now the still evolving Google Home voice-activated speaker system with Google Assistant, are only the last step providing the natural language voice interface to a modern smart home.  The first steps have to do with the protocols and connectivity of the various pieces that make a home smart, and for some protocols, a needed hub for managing them.

The basic goal of home automation boils down to providing a way to control power plugs, switches, cameras, door locks, water or gas valves, garage doors and so on using a human interface, The interface can be a wall panel, smart phone, computer, or with Alexa or Google Home, your voice.

Before I try to describe the days of fun I have had making my home smart, here is an introduction to the leading technology groups in the very nascent, volatile and confusing market today where numerous manufacturers are trying to grab a toe-hold.

Thankfully, as I found out, the trick is to choose one widely supported public open protocol and build your smart home device network using devices and hubs that support that protocol (I chose Z-Wave).

WI-FI

Since all modern homes have at least one wi-fi router serving home internet across the house, wi-fi based home-automation devices provide a simple and natural way to bring smartness in. If you have a plug-in lamp that you want to make smart, simply get a Wi-Fi enabled smart plug, connect the smart plug to your wireless network, plug in your lamp to the smart plug, and install and launch an app on your smart phone - you are done. If you have a Amazon Alexa or Google Home also connected to the same Wi-Fi network, and selected carefully, chances are excellent  there is an Alexa "skill" or Google Home app that will provide voice control to your lamp now.

There also are a myriad of Wi-Fi Smart Home devices, including Wi-Fi Bulbs that you screw into regular bulb sockets. These bulbs have built-in wireless networking capability. There are Wi-Fi enabled door locks, IR (infrared remote control) transmitters to control your TV and electronics, and so on. If you decide to go down the Wi-Fi route, you don't need any extra hardware other than a smart phone, and most can also interface with Alexa.

Belkin sells an entire range of Wi-Fi home automation devices called WeMo that by themselves are enough for a basic smart home controlled by a smart-phone app with no hub required.

Philips sells the Philps Hue family of color-changing bulbs and hubs that are controllable via voice through Amazon Echo's Alexa assistant. These are popular because of the simplicity of usage and installation.

Due to bandwidth requirements, wi-fi is required for security cameras, connected to the cloud or not. If your smart home will include cameras, a wi-fi network should be available to it for streaming videos or monitoring still images. For example, Netgear Arlo and Ring Wi-Fi Enabled Video Doorbell are directly supported by both the Wink Hub 2 and 2nd generation Samsung SmartThings smart home automation hubs.

BLUETOOTH

Bluetooth is indeed another option to control devices making up your smart home. You can control these devices directly using bluetooth capability of your smart phone. Alternatively, modern smart home hubs including the Wink Hub 2 and Samsung SmartThings 2 can interface with bluetooth devices, making them controllable from the corresponding smartphone app or over spoken commands to Amazon Alexa.

Bluetooth enabled smart home devices are easily available as well, including popular LED Light Bulbs, Power Plugs, Door Locks, Wall Switches and so on, even button pushers and blind openers.


Z-WAVE and Z-WAVE PLUS

My recommendation is to use a Z-Wave compatible hub like Wink Hub 2 and SmartThings with Z-Wave or Z-Wave Plus devices, with security cameras that use Wi-Fi. This way you will achieve centralized control with a single app on your smartphone, and voice command support via Amazon Echo's Alexa. An additional advantage of using dedicated hubs is the built-in support for the numerous protocols and standards that today's home automation devices use. For example, the Wink Hub 2 speaks Bluetooth LE, ZigBee, Z-Wave, Kidde smart smoke detectors and alarms, and Lutron Clear Connect wireless dimmers, switches and control panels, and of course, Wi-Fi.

Some confusion is created by businesses trying to lock customers into their own branded products despite using open protocols. An example is the GE Z-Wave smart switches I bought from Lowes - each package was labelled "GE Iris" since the Iris smart-hub is marketed by Lowes. It is a mistake to think these Lowes-special "GE Iris" switches work with the Iris hub only; they are in fact regular GE Z-Wave switches that work fine with my Wink Hub 2. In theory, any Z-Wave hub - be it Pulse, Trane, Wink, Nexia, Honeywell, HomeSeer, Smart Security, Harmony, Vera, Connect, Iris or SmartThings etc., should be able to talk to any Z-Wave device.

The reasons for my bias towards Z-Wave are:

  • Z-Wave was designed grounds-up for home automation
  • Most Z-Wave devices include a Z-Wave relay that work as range extenders. The Z-Wave protocol itself supports up to 4 hops, and with a range of just under 100 feet for each hop, this gives Z-Wave plenty of range
  • Mesh network protocol architecture allows two Z-Wave nodes that cannot see each other to communicate via a 3rd node, automatically
  • Z-Wave uses the  908.42 MHz frequency band that is separate from Wi-Fi and Bluetooth radio frequencies. This is a big advantage given the huge number of Wi-Fi and Bluetooth devices competing for bandwidth in a typical home, far more worse in a multi-dwelling environment like apartments and condominiums.
  • Z-Wave is regulated by a group of highly respected well-known corporations
  • Security: Z-Wave includes security and encryption that makes devices resistant to hacking
  • Device availability: At the time of writing, the Z-Wave web site claims "over 1,700 certified inter-operable products worldwide". Numerous Z-Wave devices for anything you might want to control or do with your home automation system can be found easily on Amazon and eBay.

ZigBee

ZigBee Logo | Supratim Sanyal's Blog
Like Z-Wave, ZigBee is a low-power low-speed mesh network protocol, with devices relaying packets to get over distance restrictions. It operates at the 915 MHz band in the United States. Like Z-Wave, it is still evolving. Originally debuting in 2003 and revised in 2006, ZigBee Pro was introduced in 2007 supporting backward compatibility with original ZigBee devices. Recent developments earlier this year include renaming of the cluster library as "Dotdot" and mechanisms of interoperation with Internet Protocol (IP) networks and the emerging Thread standard for connected homes. Support for Thread based home automation is built into Google Home.

There are numerous ZigBee devices available widely, and ZigBee is supported natively along with Z-Wave and Wi-Fi by both Wink Hub 2 and SmartThings smart hubs.

Among many offerings, the Osram Sylvania Lightify family of smart tunable lights and hubs uses ZigBee.


The Sanyal Family DIY Smart Home Project


The Hub - Wink Hub 2


Wink Hub 2 | Supratim Sanyal's Blog
Wink Hub 2

A home automation hub is at the center of any smart home - it is the piece of hardware that talks to all the devices making up the smart home, as well as to other gadgets adding on to control and command features, like voice command and control via Amazon Echo Alexa.

After studying the myriad home automation products and protocols and deciding to build my smart home network using the Z-Wave Plus protocol, there really were two choices for the hub: Wink Hub 2 and the Samsung SmartThings 2nd Generation.

I picked up a Wink Hub 2 because (a) it supports Z-Wave Plus, and a bunch of other protocols that I might need to use later and (b) the mobile app is well thought out, intuitively designed for regular tasks, and looks nice, and (c) it is not a Korean product.

Wink Hub 2 as shown on the Wink App on iOS iPad | Supratim Sanyal's Blog
Wink Hub 2 as shown on the Wink App on iOS (iPad)

I am very happy with the performance of the Wink Hub 2 - it paired with all of my Z-Wave devices successfully, works flawlessly and is very stable.

There was only one instance of it going into some sort of a recovery mode (flashing blue LED) and would not get out of it even on factory resetting. The solution that I found in the Wink Hub forum was to logout and log back in to the mobile app - that actually fixed it!


Smart Z-Wave Garage Door Opener: Linear Z-Wave Garage Door Remote Controller



GoControl Linear Z-Wave Smart Garage Door Opener Remote Controller | Supratim Sanyal's Blog
GoControl Linear Z-Wave Smart Garage Door Opener Remote Controller
Our garage door is opened and closed by a regular Black and Decker Craftsman garage door opener (the Craftsman brand was taken over from Sears by Black and Decker in March 2017). Instead of replacing the entire Craftsman unit with a popular and well-marketed Chamberlain smart wi-fi garage door controller which would also mean an additional phone app just for the garage door, and since I already had a Z-Wave compatible Wink Hub 2 online, I decided to install a Linear Z-Wave Garage Door Controller and connect it to the existing Craftsman motor. The same controller is also marketed as GoControl/Linear Z-Wave Garage Door Opener Remote Controller.

Connecting the Linear Z-Wave Garage Door Smart Controller to the vanilla Craftsman garage door opener | Supratim Sanyal's Blog
Connecting the Linear Z-Wave Garage Door Smart Controller to the vanilla Craftsman garage door opener

It turned out this was a pretty easy thing to do. I first paired the Linear Z-Wave Garage Door Remote Controller with my Wink Hub 2, and then mounted it with the included mounting bolts and hardware. To interface to the existing Craftsman unit, I piggybacked the two wires from the Linear controller to the same push-in connectors which connect the existing wired garage door switch. This was made simple by the push-in connectors - pressing on the red tabs released the existing wire, and I just pushed two wires twisted together back into each of the two connectors on the motor. The Linear packaging also includes all required bolts, nuts and hardware for multiple options of mounting it.


The GoControl / Linear Z-Wave Garage Door Remote Controller Position Sensor

I also screwed in the door position sensor as instructed in the manual. This sensor is battery powered, and senses if the garage door is open or closed, communicating wirelessly with the main Linear unit. It is also required for the Linear unit to operate, due to safety reasons.

Wink Hub 2 Gocontrol Smart Garage Door Remote Control App on iOS iPad | Supratim Sanyal's Blog
Wink Hub 2 Gocontrol Smart Garage Door Remote Control App on iOS
After connecting everything, the Wink app asked the door to be opened and closed once the first time to configure itself. From then on, we have been happily swiping our fingers up and down on the screens of our phones to open, close and receive notifications about our smart garage door. I have even set up a Wink robot to turn the foyer, kitchen and living room lights on if the garage door is opened after sunset on weekdays, so that my wife walks into a house with the lights already on when she returns from work.

Z-Wave Smart Switches, Dimmers and 3-Way Switches: GE Z-Wave Smart Switches


[WIP]

Saturday, January 21, 2017

DNS Amplification Attacks On Open Recursive DNS Server Running dnsmasq

I run public ad-blocking and malware/ransomware-safe recursive DNS servers for the benefit of anyone wanting to use them. These DNS servers are available at the IP addresses 64.137.248.161, 64.137.248.212 and 64.137.228.122.

I use dnsmasq with domain blacklists to block advertising, malware and ransomware URLs for clients using these DNS servers. Since they are open to the internet for public access, it did not take long for weirdos to use them for DNS amplification attacks on these servers. Here is an example of a rather hilarious DNS amplification attack logged by one of the virtual cloud servers I maintain, on new year's day of 2017, possibly originating from a virtual private server sold by Phoenix, Arizona based Nobis Technology Group now owned by LeaseWeb according to their web-site. I cannot but smile on whoever set up the DNS responder for enlansg,com.

Jan  1 07:03:05 wbri dnsmasq[23937]: query[ANY] enlansg.com from 23.82.61.2
...
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group of churches to help rebuild communities devastated by the closure of local steel plants.
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group of churches to helpff4
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is President Obama's years of public service are based around his unwavering belief in the abilit
y to unite people around a politics of purpose. In the Illinois State Senate, he passed the first major ethics reform in 25 years, cut taxes for working
 families
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is , and expanded health care for children and their parents. As a United States Senator
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is Aft3er working his way through college with the help of scholarships and student loans, Presid
ent Obama moved to Chicago, where he worked with a group of churches to help
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is With a father from Kenya and a mother from Kansas, President Obama was born in Hawaii on Augus
t 4, 1961. He was raised with help from his grandfather
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group o45f churches to help
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is he reached across the aisle to pass groundbreaking lobbying reform, lock up the world's most d
angerous weapons, and bring transparency to government by putting federal spending online.
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is President Obama's years of public service are based around his unwavering belief in the abilit
y to unite people around a politics of purpose. In the Illinois State Senate, he passed the first major ethics reform in 25 years, cut taxes for working
 families
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is , and expanded health care for children and their parents. As a United States Senator, he reac
hed across the aisle to pass groundbreaking lobbying reform, lock up the world's most dangerous weapons, and bring transparency to government by putting
 federal
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is spending online.
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is Aft1er working his way through college with the help of scholarships and student loans, Presid
ent Obama moved to Chicago, where he worked with a group of churches to help
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group of churches to help3
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is he reached across the aisle to pass groundbreaking lobbying reform, lock up the world's
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group of churches to helpff
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a g2roup of churches to help
Jan  1 07:03:05 wbri dnsmasq[23937]: reply enlansg.com is After working his way through college with the help of scholarships and student loans, Preside
nt Obama moved to Chicago, where he worked with a group of churches to help

Jan  1 07:38:17 yiradio dnsmasq[1184]: query[ANY] enlansg.com from 23.82.61.2

However, there indeed are much more sinister attackers out there, like this one originating from a source that GeoIPs to Telecom Italia, using my servers as amplifiers to attack root DNS servers and the Western Area Power Administration of the United States Government - "one of four power marketing administrations within the U.S. Department of Energy", at rates over 20 queries per second:

Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:41 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:42 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:47:42 wbri dnsmasq[23937]: query[ANY] . from 195.22.214.65
Jan  1 09:52:16 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:16 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:16 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65
Jan  1 09:52:17 wbri dnsmasq[23937]: query[ANY] wapa.gov from 195.22.214.65

So the question is what do I do about it. There is a lot of advice on the internet on why I should not run a open recursive DNS resolver in the first place, but that is exactly what I want to do. The next best thing would be to try to handle automated floods of queries, which brought me to a nice little post by Matteo CastelliBlocking DNS Amplification attacks.

How many DNS queries is it normal for a web-browser to send out for an average web page? Over 20 seems to me to be unlikely, especially for a caching server like dnsmasq. Based on a pure gut feeling, with no scientific analysis whatsoever, I decided 10 is a good number.

Following Matteo's work, I replaced the line that allows DNS requests in my CentOS 7 iptables configuration file /etc/sysconfig/iptables to limit the number of queries per second. 

# -- OLD -- -A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -m limit --limit 10/sec -j LOG --log-prefix "fw-dns " --log-level 7

This takes effect on reboot. To update the rule dynamically on a running system, I usually use iptables-save > tempfile, edit tempfile and then iptables-restore < tempfile.

This firewall tweak hopefully allows reasonable use of my open recursive DNS resolvers by web browsers while limiting damage by amplification attackers. If this is not enough, I will configure fail2ban also as recommended by Matteo. I am open to other ideas as long as they do not call for me to not run an internet-facing resolver.


Thursday, January 12, 2017

Mount Windows NTFS format external USB hard disk on Solaris 11 (SunOS 5.11 - Solaris 11.3)

mount ntfs on solaris 11 using ntfs-3g

Introduction

This post applies to openindiana Community-driven illumos Distribution only. To do this on the official Oracle Solaris 11.3 kernel please see my post on Oracle Solaris 11.3.


STEP 1


Install ntfs-3g with ntfsprogs and fuse file system packages.

The ntfs-3g related packages at SFE are old, buggy and crash-prone, causing kernel panics every time I tried to share my external NTFS drive over a Samba share. Do not intsall these from SFE. Rather, install the updated debugged packages for Solaris 11 from the NTFS-3G on OpenIndiana page which includes spot-on instructions.

Here is a quick summary of the steps to install ntfs-3g with ntfsprogs and fuse on 64-bit Solaris 11 on Intel processors from the above webpage:

a) Use wget or curl to download ntfs-3g_ntfsprogs-2016.2.22AR.2.pkg64.zip

b) If ntfs-3g was already installed :
pkg uninstall ntfs-3g

c) unzip -x ntfs-3g_ntfsprogs-2016.2.22AR.2.pkg64.zip

d) pkgadd -d ntfs-3g_ntfsprogs-2016.2.22AR.2.pkg64 all

Note: If a warning is displayed saying "The following files are already installed on the system and are being used by another package:" followed by the question "Do you want to install these conflicting files [y,n,?,q]", answer y for "yes" to install the conflicting files from the current package:

Processing package instance <ntfs-3g> from </export/home/xxxxx/ntfs-3g_ntfsprogs-2016.2.22AR.2.pkg64>

ntfs-3g(x86_64) 2016.2.22AR.2
## Processing package information.
## Processing system information.
   11 package pathnames are already properly installed.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.

The following files are already installed on the system and are being
used by another package:
* /usr/sbin/mkntfs
* /usr/sbin/ntfsclone
* /usr/sbin/ntfslabel
* /usr/sbin/ntfsresize
* /usr/sbin/ntfsundelete

* - conflict with a file which does not belong to any package.

Do you want to install these conflicting files [y,n,?,q] y
## Checking for setuid/setgid programs.

Installing ntfs-3g as <ntfs-3g>

## Installing part 1 of 1.
/usr/bin/lowntfs <symbolic link>
/usr/bin/lowntfs-3g
/usr/bin/ntfs <symbolic link>
/usr/bin/ntfs-3g
/usr/bin/ntfs-3g.probe
/usr/bin/ntfsfix
/usr/bin/ntfsinfo
/usr/bin/ntfssecaudit
/usr/bin/ntfsusermap
/usr/bin/ntfswipe
/usr/lib/amd64/libntfs-3g.so <symbolic link>
/usr/lib/amd64/libntfs-3g.so.872 <symbolic link>
/usr/lib/amd64/libntfs-3g.so.872.0.0
/usr/lib/fs/lowntfs-3g/fstyp
/usr/lib/fs/lowntfs-3g/mount <symbolic link>
/usr/lib/fs/lowntfs/fstyp <symbolic link>
/usr/lib/fs/lowntfs/mount <symbolic link>
/usr/lib/fs/ntfs-3g/fstyp
/usr/lib/fs/ntfs-3g/mount <symbolic link>
/usr/lib/fs/ntfs/fstyp <symbolic link>
/usr/lib/fs/ntfs/mount <symbolic link>
/usr/lib/pkgconfig/libntfs-3g.pc
/usr/sbin/mkntfs
/usr/sbin/ntfsclone
/usr/sbin/ntfslabel
/usr/sbin/ntfsresize
/usr/sbin/ntfsundelete
/usr/share/doc/ntfs-3g/README
/usr/share/man/man8/mkntfs.8
/usr/share/man/man8/mount.lowntfs-3g.8 <symbolic link>
/usr/share/man/man8/mount.ntfs-3g.8 <symbolic link>
/usr/share/man/man8/ntfs-3g.8
/usr/share/man/man8/ntfsclone.8
/usr/share/man/man8/ntfsfix.8
/usr/share/man/man8/ntfsinfo.8
/usr/share/man/man8/ntfslabel.8
/usr/share/man/man8/ntfsresize.8
/usr/share/man/man8/ntfssecaudit.8
/usr/share/man/man8/ntfsundelete.8
/usr/share/man/man8/ntfsusermap.8
/usr/share/man/man8/ntfswipe.8
[ verifying class <System> ]

Installation of <ntfs-3g> was successful.

e) If a fuse kernel module was already installed :
pkg uninstall libfuse
pkg uninstall fusefs

reboot

f) Use wget or curl to download fuse-kernel-1.2AR.7.pkg.zip

g) unzip -x fuse-kernel-1.2AR.7.pkg.zip

h) pkgadd -d fuse-kernel-1.2AR.7.pkg all

Watch the output for something like this:


Processing package instance <fusefs> from </export/home/username/fuse-kernel-1.2AR.7.pkg>

fusefs(x86) 1.2AR.7
## Processing package information.
## Processing system information.
   4 package pathnames are already properly installed.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

Installing fusefs as <fusefs>

## Installing part 1 of 1.
/usr/kernel/drv/amd64/fuse
/usr/kernel/drv/fuse
/usr/kernel/drv/fuse.conf
[ verifying class <System> ]

Installation of <fusefs> was successful.


i) If this is the first time a fuse kernel module is being installed:
add_drv -m 'fuse 0666 root sys' fuse
ln -s /devices/pseudo/fuse@0:fuse /dev/fuse

j) Reboot Solaris 11:
reboot

STEP 2

Find connected external USB drive and mount it using ntfs-3g.

a) Use the command

cfgadm | fgrep conn

to verify USB storage device is seen by Solaris 11 and connected, as in usb0/2 in the following example:

root@solaris11-3:~# cfgadm | fgrep conn
usb1/1                         usb-storage  connected    configured   ok


b) Use the command:

# rmformat

to get the raw disk device name for your external USB HDD, as in /dev/rdsk/c3t0d0p0 in the example below:

root@solaris11-3:~# rmformat
Looking for devices...
     1. Logical Node: /dev/rdsk/c1t1d0p0
        Physical Node: /pci@0,0/pci-ide@1,1/ide@0/sd@1,0
        Connected Device: VBOX     CD-ROM           1.0
        Device Type: DVD Reader
        Bus: IDE
        Size: <Unknown>
        Label: <Unknown>
        Access permissions: <Unknown>
     2. Logical Node: /dev/rdsk/c3t0d0p0
        Physical Node: /pci@0,0/pci8086,265c@b/storage@1/disk@0,0
        Connected Device: WD       My Book 1110     1030
        Device Type: Removable
        Bus: USB
        Size: 1430.1 GB
        Label: <Unknown>
        Access permissions: <Unknown>
     3. Logical Node: /dev/rdsk/c3t0d1p0
        Physical Node: /pci@0,0/pci8086,265c@b/storage@1/disk@0,1
        Connected Device: WD       Virtual CD 1110  1030
        Device Type: CD Reader
        Bus: USB
        Size: 668.0 MB
        Label: <None>
        Access permissions: <Unknown>

c) Confirm the device name (excuding the partition number) using the iostat -En command. For example, for an entry for /dev/rdsk/c3t0d0p0 returned by the rmformat command above, iostat -En should show the device c3t0d0:

# iostat -En
c1d0             Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Model: VBOX HARDDISK   Revision:  Serial No: VB4e58aa27-8952 Size: 33.55GB <33553981440 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0
c1t1d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: VBOX     Product: CD-ROM           Revision: 1.0  Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0 Non-Aligned Writes: 0
c3t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 1
Vendor: WD       Product: My Book 1110     Revision: 1030 Serial No:
Size: 1499.60GB <1499598946304 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 45 Predictive Failure Analysis: 0 Non-Aligned Writes: 0
c3t0d1           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: WD       Product: Virtual CD 1110  Revision: 1030 Serial No:
Size: 0.70GB <700448768 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 13 Predictive Failure Analysis: 0 Non-Aligned Writes: 0



d) Figure out (maybe also write down) the device name to use with ntfs-3g. To do this, take the device name returned by rmformat in the previous step (/dev/rdsk/c3t0d0p0 in my case), and
  • replace rsdk with dsk
  • replace partition identifier 0 with 1 at the end, e.g. c3t0d1p0 with c3t0d1p1. I am not sure if this is needed because the Western Digital My Book USB HDD I am using has a virtual CD-ROM partition as well for WD Smartware; this partition is however successfully auto-mounted by Solaris 11.
As an example, for the rmformat reported raw disk device name /dev/rdsk/c3t0d0p0 which has only one partition, the device name to use with ntfs-3g is /dev/dsk/c3t0d0p1.

e) Mount the device identified in the prior step using the ntfs-3g command, like so:

# ntfs-3g  /dev/dsk/c3t0d0p1  /media
The disk contains an unclean file system (0, 0).
The file system wasn't safely closed on Windows. Fixing.

Your NTFS format external USB hard disk is now mounted at the mount-point /media on your Solaris 11 system.



STEP 3

Make it permanent - mount at boot.

I first tried adding a  line to /etc/vfstab like so (note the device name with rdsk, not dsk, for the 2nd field, i.e. device to fsck):

/dev/dsk/c3t0d0p1 /dev/rdsk/c3t0d0p1 /media ntfs-3g - yes -

but it does not work! (I am not sure why.) The external USB drive is not mounted at boot time using the above line in vfstab, so I backed out and tried a more elaborate approach.

I created a file /etc/rc.local with the following contents

# ---
# Commands to execute at end of boot
# This is a linked from /etc/rc3.d/S99local
# Solaris 11 still supports this
# ---
/usr/bin/ntfs-3g /dev/dsk/c3t0d0p1 /media

and then placed a symbolic link from /etc/rc3.d/S99local to /etc/rc.local.

# chmod +x /etc/rc.local
# ln -s /etc/rc.local /etc/rc3.d/S99local
# ls -l /etc/rc.local /etc/rc3.d/S99local
-rwxr-xr-x   1 root     root         357 Jan 23 19:30 /etc/rc.local
lrwxrwxrwx   1 root     root          13 Jan 20 19:12 /etc/rc3.d/S99local -> /etc/rc.local


This worked. On reboot, the external USB drive came up mounted on /media.

root@sanyalnet-solaris:~# mount
/ on rpool/ROOT/solaris read/write/setuid/devices/rstchown/dev=4750002 on Thu Jan  1 00:00:00 1970
/devices on /devices read/write/setuid/devices/rstchown/dev=8a80000 on Thu Jan 12 21:43:39 2017
/dev on /dev read/write/setuid/devices/rstchown/dev=8ac0000 on Thu Jan 12 21:43:39 2017
/system/contract on ctfs read/write/setuid/devices/rstchown/dev=8b80001 on Thu Jan 12 21:43:39 2017
/proc on proc read/write/setuid/devices/rstchown/dev=8b00000 on Thu Jan 12 21:43:39 2017
/etc/mnttab on mnttab read/write/setuid/devices/rstchown/dev=8bc0001 on Thu Jan 12 21:43:39 2017
/system/volatile on swap read/write/setuid/devices/rstchown/xattr/dev=8c00001 on Thu Jan 12 21:43:39 2017
/system/object on objfs read/write/setuid/devices/rstchown/dev=8c40001 on Thu Jan 12 21:43:39 2017
/etc/dfs/sharetab on sharefs read/write/setuid/devices/rstchown/dev=8c80001 on Thu Jan 12 21:43:39 2017
/lib/libc.so.1 on /usr/lib/libc/libc_hwcap1.so.1 read/write/setuid/devices/rstchown/dev=4750002 on Thu Jan 12 21:43:56 2017
/dev/fd on fd read/write/setuid/devices/rstchown/dev=8d80001 on Thu Jan 12 21:43:56 2017
/var on rpool/ROOT/solaris/var read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=4750004 on Thu Jan 12 21:44:07 2017
/tmp on swap read/write/setuid/devices/rstchown/xattr/dev=8c00002 on Thu Jan 12 21:44:07 2017
/var/share on rpool/VARSHARE read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=4750005 on Thu Jan 12 21:44:08 2017
/export on rpool/export read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=4750006 on Thu Jan 12 21:44:18 2017
/export/home on rpool/export/home read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=4750007 on Thu Jan 12 21:44:19 2017
/rpool on rpool read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=4750009 on Thu Jan 12 21:44:22 2017
/system/zones on rpool/VARSHARE/zones read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=475000a on Thu Jan 12 21:44:22 2017
/var/share/pkg on rpool/VARSHARE/pkg read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=475000b on Thu Jan 12 21:44:23 2017
/var/share/pkg/repositories on rpool/VARSHARE/pkg/repositories read/write/setuid/devices/rstchown/nonbmand/exec/xattr/atime/dev=475000c on Thu Jan 12 21:44:24         2017
/media on /devices/pci@0 read/write/nosetuid/nodevices/rstchown/dev=4a00000 on Thu Jan 12 21:48:09 2017
root@sanyalnet-solaris:~#