Fishing for depth data

Something in the wind …

I have mined data on hafro directories and obtained this:

q <- 
  arrow::open_dataset("/net/hafkaldi.hafro.is/export/home/haf/einarhj/stasi/gis/botninn/data-parquet/xyz")
q |> dplyr::glimpse()
FileSystemDataset with 117 Parquet files
1,012,889,718 rows x 8 columns
$ x      <double> 1485135, 1485130, 1485222, 1485229, 1485193, 1485235, 1485183,…
$ y      <double> 391204.1, 391210.8, 391137.4, 391147.5, 391160.4, 391160.3, 39…
$ z      <double> -89.258, -89.121, -86.813, -87.836, -88.985, -87.769, -88.950,…
$ sq      <int32> 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17…
$ file   <string> "Arnarfj_03.xyz", "Arnarfj_03.xyz", "Arnarfj_03.xyz", "Arnarfj…
$ area   <string> "Arnarfjördur", "Arnarfjördur", "Arnarfjördur", "Arnarfjördur"…
$ pth    <string> "data-copy/LHG/LHG_Vestf/sjom_2013/Arnarfj_03.xyz", "data-copy…
$ source <string> "lhg", "lhg", "lhg", "lhg", "lhg", "lhg", "lhg", "lhg", "lhg",…

So some putative ~1000 millions individual pings, albeit though that some may actually have been rasterized a priori.

The coverage:

dx <- 500
d <- 
  arrow::open_dataset("/net/hafkaldi.hafro.is/export/home/haf/einarhj/stasi/gis/botninn/data-parquet/xyz") |> 
  dplyr::mutate(x = x %/% dx * dx + dx/2,
                y = y %/% dx * dx + dx/2) |> 
  dplyr::group_by(x, y) |> 
  dplyr::summarise(n = n(),
                   .groups = "drop") |> 
  dplyr::collect()
d |> 
  dplyr::mutate(n = ifelse(n > 1e4, 1e4, n)) |> 
  dplyr::filter(x < 2e6) |> 
  ggplot2::ggplot(ggplot2::aes(x, y, fill = n)) +
  ggplot2::theme_void() +
  ggplot2::geom_tile() +
  ggplot2::scale_fill_viridis_c() +
  ggplot2::coord_equal() +
  ggplot2::labs(fill = "pings per\n500x500 meters")

The above includes some archieved data from MFRI, the original having these number of records and I must say intriguing variables:

arrow::open_dataset("/net/hafkaldi.hafro.is/export/home/haf/einarhj/stasi/gis/botninn/data-parquet/XYZ_Oracle_gagnagrunnur") |> 
  dplyr::glimpse()
FileSystemDataset with 25 Parquet files
347,205,875 rows x 21 columns
$ x          <double> 1453843, 1453834, 1453825, 1453816, 1453808, 1453797, 1453…
$ y          <double> 518175.9, 518170.0, 518164.8, 518159.1, 518154.1, 518147.5…
$ z          <double> -231.64, -230.48, -229.80, -229.01, -228.67, -227.10, -226…
$ sq          <int32> 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10…
$ file       <string> "Djupall_2002_byte8.txt", "Djupall_2002_byte8.txt", "Djupa…
$ area       <string> "Djúpáll", "Djúpáll", "Djúpáll", "Djúpáll", "Djúpáll", "Dj…
$ pth        <string> "data-copy/XYZ_Oracle_gagnagrunnur/2002/Djupall_2002_byte8…
$ date       <string> "2002-183", "2002-183", "2002-183", "2002-183", "2002-183"…
$ time       <string> "02:27:27.350", "02:27:27.350", "02:27:27.350", "02:27:27.…
$ project    <string> "Djupall", "Djupall", "Djupall", "Djupall", "Djupall", "Dj…
$ vessel     <string> "ArniFr", "ArniFr", "ArniFr", "ArniFr", "ArniFr", "ArniFr"…
$ line       <string> "0012_20020702_022724_raw", "0012_20020702_022724_raw", "0…
$ profile     <int32> 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…
$ beam        <int32> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,…
$ tide       <double> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ accuracy    <int32> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ status     <string> "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A"…
$ amp_db      <int32> -32, -32, -31, -31, -31, -31, -30, -30, -30, -30, -31, -30…
$ amp_byte    <int32> 191, 191, 193, 193, 194, 194, 195, 195, 195, 195, 194, 196…
$ alng_angle <double> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0…
$ arc_angle  <double> -66.70, -66.43, -66.15, -65.85, -65.55, -65.24, -64.93, -6…

Now this is what I will call the basic working space when it comes to operating with bathymetry data, whatever the post-processing software. For next steps using these type of data within R check out A workflow model.

A full list of the source, area and original filename:

arrow::open_dataset("/net/hafkaldi.hafro.is/export/home/haf/einarhj/stasi/gis/botninn/data-parquet/xyz") |> 
  dplyr::count(source, area, file) |> 
  dplyr::collect() |> 
  dplyr::arrange(source) |> 
  knitr::kable()
source area file n
haf Halinn Halinn_2005_byte8.txt 13569134
haf Halinn Halinn_2007_byte8.txt 81677708
haf Halinn Halinn_2002_byte.txt 17214112
haf Kolluáll Kolluall_2011_byte8.txt 7145900
haf Hryggur Hryggur_2011_byte.txt 43315636
haf Langanes NA_fridur_byte8.txt 26293805
haf Víkuráll Vikurall_2002_byte.txt 16004124
haf Ísafjarðardjúp Isafjardardjup_2003_byte8.txt 647257
haf Ísafjarðardjúp Isafjardardjup_2004_byte8.txt 2227658
haf Ísafjarðardjúp Isafjardardjup_2008_byte8.txt 54737839
haf Halinn Halinn_2008_byte8.txt 10944076
haf Kolluáll Kolluall_2008_byte8.txt 12433158
haf Reykjaneshryggur Reykjaneshryggur_2003_byte8.txt 1657990
haf Reykjaneshryggur Reykjaneshryggur_2004_byte8.txt 4381947
haf Víkuráll Vikurall_2005_byte.txt 18294926
haf Ísafjarðardjúp Isafjardardjup_2002_byte8.txt 4211109
haf Djúpáll Djupall_2005_byte8.txt 1348094
haf Víkuráll Vikurall_2008_byte.txt 3197108
haf Ísafjarðardjúp Isafjardardjup_2005_byte8.txt 6333166
haf Djúpáll Djupall_2003_byte8.txt 5727076
haf Ísafjarðardjúp Isafjardardjup_2010_byte8.txt 969445
haf Djúpáll Djupall_2002_byte8.txt 4249634
haf Reykjaneshryggur Reykjaneshryggur_2002_byte8.txt 1300958
haf Halinn Halinn_2003_byte.txt 5685951
haf Víkuráll Vikurall_2007_byte.txt 3638064
lhg Breiðafjörður k426_dypi_allt.txt 465189
lhg Breiðdalsvík Brdvik_north_bin3.txt 2713658
lhg Dýrafjörður Dyrafj_01_bin2.txt 3633306
lhg Dýrafjörður Dyrafj_02_bin2.xyz 11550865
lhg Dýrafjörður Dyrafj_03_bin4.txt 4782701
lhg Dýrafjörður Dyrafj_03_bin4.xyz 4782701
lhg Flatey flatey_5m_depthLL.xyz.gz 727841
lhg Fáskrúðsfjörður Fa_fjord2_bin3.txt 4442218
lhg Fáskrúðsfjörður Fa_fjord2_bin1.txt 21282132
lhg Gjögurtá gjogurta_5m_depthLL.xyz.gz 2911873
lhg Ísland LHG_single_beam.csv 28462631
lhg Mjóifjörður Mj_fjord2_bin1.txt 8436123
lhg Loðmundarfjörður Lo_fjord_bin1.txt 18813828
lhg Norðfjörður No_fjord1_bin1.txt 16850199
lhg Mjóifjörður Mj_fjord3_bin1.txt 18416306
lhg Norðfjörður No_fjord2_bin1.txt 21464222
lhg Norðfjörður No_fjord_bin1.txt 1877368
lhg Reyðarfjörður R_fjord2_bin3.txt 3336907
lhg Reyðarfjörður R_fjord2_bin1.txt 13427887
lhg Reyðarfjörður R_fjord1_bin1.txt 18326897
lhg Reyðarfjörður R_fjord3_bin1.txt 17091750
lhg Reyðarfjörður R_fjord4_bin1.txt 24897353
lhg Reyðarfjörður R_fjord_bin1.txt 6928833
lhg Reyðarfjörður R_fjord5_bin3.txt 4707727
lhg Seyðisfjörður Sey_fjord1_bin1.txt 4120946
lhg Stöðvarfjörður St_fjord2_bin1.txt 22513671
lhg Stöðvarfjörður St_fjord_bin1.txt 13909386
lhg Vaðlavík Vadlavik_bin3.txt 4115737
lhg Vaðlavík Vadlavik_bin1.txt 18062143
lhg Vestmannaeyjar Sv-212_bin2.txt 11370217
lhg Vestmannaeyjar Sv-215_bin2.txt 14035956
lhg Víkur Vikur_south_bin1.txt 19322548
lhg Víkur Vikur_north_bin1.txt 20000302
lhg Breiðdalsvík Brdvik_north_bin1.txt 17091253
lhg Dýrafjörður Dyrafj_01_bin2.xyz 3633306
lhg Dýrafjörður Dyrafj_02_bin2.txt 11550865
lhg Fáskrúðsfjörður Fa_fjord3_bin1.txt 8826691
lhg Mjóifjörður Mj_fjord3_bin3.txt 4308949
lhg Norðfjörður No_fjord2_bin3.txt 5929424
lhg Patreksfjörður Patro_01_bin3.txt 1872120
lhg Patreksfjörður Patro_02_bin3.txt 6820928
lhg Patreksfjörður patro_bin3.xyz 2133637
lhg Patreksfjörður patro_03.xyz 3888896
lhg Sandvík Sandvik_bin1.txt 18611225
lhg Seyðisfjörður Sey_fjord3_bin3.txt 2953624
lhg Seyðisfjörður Sey_fjord2_bin1.txt 12046553
lhg Súgandafjörður Sugandafj_00_bin4.xyz 1894185
lhg Tálknafjörður Talkni_03_bin3.txt 4679555
lhg Vestmannaeyjar Sv-216_bin2.txt 11705876
lhg Vestmannaeyjar Sv-213_bin2.txt 10543514
lhg Önundarfjörður Onundafj_01_bin4.xyz 2544762
lhg Fáskrúðsfjörður Fa_fjord3_bin3.txt 2055614
lhg Mjóifjörður Mj_fjord1_bin1.txt 4145668
lhg Patreksfjörður Patro_03_bin3.txt 3888896
lhg Patreksfjörður Patro_bin3.txt 2133637
lhg Patreksfjörður patro_01.xyz 1872120
lhg Reyðarfjörður R_fjord5_bin1.txt 16334224
lhg Sandvík Sandvik_bin3.txt 4070082
lhg Vestmannaeyjar Sv-214_bin2.txt 10591877
lhg Flóinn floinn_5m_depthLL.xyz.gz 1104304
lhg Norðfjörður No_fjord1_bin3.txt 3031928
lhg Reyðarfjörður R_fjord1_bin3.txt 3247371
lhg Seyðisfjörður Sey_fjord3_bin1.txt 14172753
lhg Vestmannaeyjar Sv-210_bin2.txt 10246813
lhg Víkur Vikur_north_bin3.txt 3292616
lhg Arnarfjördur Arnarfj_04.xyz 1313432
lhg Fáskrúðsfjörður Fa_fjord1_bin1.txt 9025031
lhg Reyðarfjörður R_fjord3_bin3.txt 3294689
lhg Reyðarfjörður R_fjord4_bin3.txt 4999961
lhg Seyðisfjörður Sey_fjord2_bin3.txt 2599082
lhg Önundarfjörður Onundafj_02_bin4.xyz 4731258
lhg Tálknafjörður Talkni_02_bin3.txt 2428691
lhg Lundey lundey_5m_depthLL.xyz.gz 1261739
lhg Patreksfjörður patro_02.xyz 6820928
lhg Vestmannaeyjar Sv-211_bin2.txt 5154681
lhg Loðmundarfjörður Lo_fjord_bin3.txt 3125672
lhg Mjóifjörður Mj_fjord1_bin3.txt 733596
lhg Mjóifjörður Mj_fjord2_bin3.txt 1670705
lhg Stöðvarfjörður St_fjord_bin3.txt 2020640
lhg Eyjafjörður Eyjafj_MultiBeam_5m.xyz 4619656
lhg Fáskrúðsfjörður Fa_fjord1_bin3.txt 1946758
lhg Seyðisfjörður Sey_fjord1_bin3.txt 787149
lhg Kolgrafarfjörður Sv-196.xyz 1077
lhg Stöðvarfjörður St_fjord2_bin3.txt 3972333
lhg Arnarfjördur Arnarfj_03.xyz 1169836
lhg Víkur Vikur_south_bin3.txt 3071588
lhg Norðfjörður No_fjord_bin3.txt 289133
lhg Eyjafjörður Eyjafj_SingleBeam_baldur_1994.txt 7960
lhg Kolgrafarfjörður Sv-197-vegagerdin.XYZ 4308
lhg Vestmannaeyjar Sv-209_bin2.txt 115849
lhg Tálknafjörður Talkni_01_bin3.txt 362410
lhg Reyðarfjörður R_fjord_bin3.txt 1145609
lhg Kolgrafarfjörður allt.XYZ 5385

The code behind how I got here is on github

A link placeholder: