There is paint that can make whatever you are painting supposedly gets a blackboard surface. Not much help for you but maybe to the interest for others.
>I understood the GP's comment to refer to the state of the second grad student. How useful is K-means clustering without knowing K in advance?
There are several heuristics for this. Googling I see that the elbow method, average sillhouette method and gap statistic method is the most used.
I think you could play around with your own heuristics as well. Simple KDE plots showing the amount of peaks. Maybe, say the variance between clusters should be greater than the variance inside any cluster could maybe work. (Edit: this seems to be the main point of the average sillhouette method).
Depending on the species. There are factory ships, such as the name suggest have onboard factories that process the fish into filets, pack, freeze and stow the fish, enabling the vessels to stay longer out at sea - limited either by their cargo capacity or quota. Arguably this is good for the quality of the fish as well since it is frozen fresh.
I have worked with some catch logging data, and there is certainly some weird stuff in some of the data that points to something like that (but super interesting to untangle and figure out, from a personal perspective). It really is a game of finding clear, understandable regulations, but at the same time put a lot of work into checking whether they are upheld.
As it is in Norway this is an ongoing debate - where the smaller fishing vessels have been exempt from having the mandatory AIS equipment, citing privacy as some of the reasons. In the government portal for showing AIS data they do not show data of ships that is below 45m length to preserve their privacy. However larger fishing vessels have large crews depending on the mode of fishing, so it becomes less a case of tracking single persons (and maybe in the fishermen more a case of tracking commercial competitors).
This is really interesting, du you mind maybe sharing some names of companies or research papers here? I am working with this topic myself in my phd, however more from the AIS perspective. While AIS is a great data source it is indeed easy to fool, all these initiatives to illuminate the dark spots are very interesting.
There's a bunch of work with sentinel sar sats and a few others tracking marine VHF and radar emissions. I can't go much into the newer efforts though unfortunately.
Thank you for your suggestions! I'm mainly looking at how we can use AIS data to gain knowledge about the maritime domain, with an emphasis on fisheries in the north (norwegian sea and the rest of the arctic) , but as there is so many interesting cases here I constantly go down side paths admittedly, especially if you pull in things like operations research and the newer advances of deep learning.
The ML behind this is most probably building on the work of Kroodsma et. al (2018) [1]. (Kroodsma is affiliated with Global Fishing Watch). While AIS data can contain information about whether a ship is engaged in fishing, this is sparsely used - even though there is no ill intent. By using spatio-temporal data such as position and speed, and expert labelled segments of activities they trained CNN's to identify fishing activity from other activities. Since these are vessels that did not broadcast their positions, i.e did not broadcast AIS data my guess is that they used the optical imagery to construct movement patterns, maybe even speeds (by looking at the wake patterns) as well as their position in general as input data to similar constructed CNN's. They could also put in info such as whether the ship was in or near fishing grounds, and whether the ship showed signs to travel to port to offload any catch, or met up with a vessel to transfer the catch.
I'm currently in the middle of my phd where I am working with these types of data and methods, its extremely interesting.
[1] David A. Kroodsma et al. ,Tracking the global footprint of fisheries.Science359,904-908(2018).DOI:10.1126/science.aao5646
nice! So like a bit of domain expert input, a lot of space time data, and maybe things like velocity. I guess the main question is what's the biggest difference between the data profile of an illegal fishing boat and a recreational boat? I'm not sure what you're filtering OUT...
Yes, a bit of that! If the fishing vessel is emitting AIS data you will already have the code from the AIS transmitter as well as the IMO number (like a social security number but for ships), and by using these numbers you can see whether it is a fishing vessel or not. These can ofcouse be spoofed in the case of illegal fishing vessels, but I think the likelihood of this happening in the terretorial waters of a country with an established coastal authority is slim.
In that case it is more about identifying the actual fishing activity than the fishing boat itself. Each mode of fishing (which requires different types of gear and types of ships) will leave a distinct signature. For example bottom-trawling will require a certain (low) speed (here a big net is dragged behind the vessel), purse seines will make somewhat circular patterns (they lay out a net in a circle around a school of fish). So often you want to identify these fishing activities to, for example link them up with a later delivery of catch, but also to investigate and estimate things like fuel usage.
Even if you don't have the IMO number of the ship you can look for these patterns, so it can be redused to whether the ship is in proximity of a fishing ground, if they display the movement patterns that is the signature of fishing activity, and later if they meet up with other vessels or go in to land to deliver their catch. A recreational boat (given that it is large enough to be suspected to be a fishing vessel just by where it is and how large it is) would perhaps have a constant high velocity (or complete stand still) for instance.
A simplified example: a person is jogging on the road that passes your apple trees you would not suspect them of stealing apples, if they are slowly walking - maybe (but if they stay no the road - probably not), if they are jogging through your garden of apple trees its certainly suspicious, but if they are slowly walking through your garden I would send a coastal guard ship ;)
A less simplified one would be to try to match up fishing related activity with how much fish a vessel is delivering (is it delivering 20 tonnes but have spent two full weeks with fishing activity?). Then maybe see if they have the right permissions to do that kind of fisheries in that zone at that time of the year etc.
1. ```{python}