Alex - Finally been able to get something working for you.. Of course need further belts and braces but let me know how this goes....

public class SendEmailwithICSAttachment {
    public static void Sendmail_With_IcsAttachment(DateTime startDate, DateTime endDate, String location, String body, String subject, 
                                                                                            String fromAddress, String displayName, String toAddress) {
        String attachmentName = 'Meeting.ics';
        Messaging.SingleEmailMessage msg = new Messaging.SingleEmailMessage();
        //Now we have to set the value to Mail message properties
 
        //Note Please change it to correct mail-id to use this in your application
        msg.setReplyTo(fromAddress);
        msg.setSenderDisplayName(displayName);
        msg.setToAddresses(new String[] {toAddress});
        //msg.setCcAddresses(new String[] {'user@acme.com'});// it is optional, only if required
        msg.setSubject(subject);
        msg.setPlainTextBody(body);

        // Create the attachment
        Messaging.EmailFileAttachment efa = new Messaging.EmailFileAttachment();
        efa.setFileName(attachmentName );
        Blob b = doIcsAttachment(startDate, endDate, location, body, subject, fromAddress, displayName, toAddress);
        efa.setBody(b);
        msg.setFileAttachments(new Messaging.EmailFileAttachment[] {efa});
        // Send the email you have created.
        Messaging.sendEmail(new Messaging.SingleEmailMessage[] { msg });
    }
    
    private static Blob doIcsAttachment(DateTime startDate, DateTime endDate, String location, String body, 
                                                                        String subject, String fromAddress, String displayName, String toAddress) {
       
        // Now Contruct the ICS file
        String [] icsTemplate = new List<String> {'BEGIN:VCALENDAR',
                                                                                'PRODID:-//Schedule a Meeting',
                                                                                'VERSION:2.0',
                                                                                'METHOD:REQUEST',
                                                                                'BEGIN:VEVENT',
                                                                                'DTSTART: ' + String.valueof(startDate),
                                                                                'DTSTAMP: '+ String.valueof(DateTime.Now()),
                                                                                'DTEND: ' + String.valueof(endDate),
                                                                                'LOCATION: ' + location,
                                                                                'UID: ' + String.valueOf(Crypto.getRandomLong()),
                                                                                'DESCRIPTION: ' + body,
                                                                                'X-ALT-DESC;FMTTYPE=text/html: ' + body,
                                                                                'SUMMARY: ' + subject,
                                                                                'ORGANIZER:MAILTO: ' + fromAddress,
                                                                                'ATTENDEE;CN="' + displayName + '";RSVP=TRUE:mailto: ' + toAddress,
                                                                                'BEGIN:VALARM',
                                                                                'TRIGGER:-PT15M',
                                                                                'ACTION:DISPLAY',
                                                                                'DESCRIPTION:Reminder',
                                                                                'END:VALARM',
                                                                                'END:VEVENT',
                                                                                'END:VCALENDAR'
                                                                                };
        String attachment = String.join(icsTemplate, '\n');
        return Blob.valueof(attachment ); 
    }
}

Just using plain text content here but can easily expand to use HTML in email and attachment.

Do mark this as a solution so that it helps other looking around.